From 87dae3567d54f4fc81b93b9a63775c1ba6ac9636 Mon Sep 17 00:00:00 2001 From: Max Date: Mon, 15 Jun 2026 02:11:52 +0530 Subject: [PATCH 01/22] Add optimize CI, and update documentation (#85) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * merge branch master (#78) * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Eamon Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Optimize CI workflow and Docker configurations with refactoring (#72) (#76) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp * docs:report [run_20260530_165216](~791 tok/s) (#60) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs: report [run_20260530_165216] (~791 tok/s) (#62) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 --------- * chore: clang-format configuration file based on LLVM (#63) * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. * Refactor core architecture and optimize CUDA features (#75) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- --------- --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor .dockerignore for improved clarity Removed unnecessary entries to streamline the build context. * Enhance .gitignore with additional exclusions Expanded .gitignore to include more file types and directories. * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add optimize CI, and update documentation (#84) * merge branch master of codeaddict (#77) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * merge branch master (#78) * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Eamon Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Optimize CI workflow and Docker configurations with refactoring (#72) (#76) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp * docs:report [run_20260530_165216](~791 tok/s) (#60) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs: report [run_20260530_165216] (~791 tok/s) (#62) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 --------- * chore: clang-format configuration file based on LLVM (#63) * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. * Refactor core architecture and optimize CUDA features (#75) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- --------- --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor .dockerignore for improved clarity Removed unnecessary entries to streamline the build context. * Enhance .gitignore with additional exclusions Expanded .gitignore to include more file types and directories. * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update main.py * Delete quadtrix_training_report.png * Delete docker-compose.yml * Delete docker-compose.gpu.yml * Delete docker-compose.dev.yml * Delete benchmark_results.csv * Delete SECURITY.md * refactor: switch tokenizer from gpt2 to tiktoken o200k * Delete contributing.md * Delete CUDA/llmcpp directory * Delete CUDA directory * Optimization (#82) * Modify training configuration parameters Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * Refactor training configuration and optimize CI workflows (#81) * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * merge branch master of codeaddict (#77) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forwa… --------- Signed-off-by: dependabot[bot] Co-authored-by: Eamon Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> From 6703c1e29d8dae75b54f781d278d91d7b12f8260 Mon Sep 17 00:00:00 2001 From: Eamon Sippy Date: Mon, 15 Jun 2026 19:52:07 +0530 Subject: [PATCH 02/22] Update training configuration parameters (#89) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Modify training configuration parameters Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * Refactor training configuration and optimize CI workflows (#81) * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * merge branch master of codeaddict (#77) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- * merge branch master (#78) * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Eamon Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Optimize CI workflow and Docker configurations with refactoring (#72) (#76) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp * docs:report [run_20260530_165216](~791 tok/s) (#60) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs: report [run_20260530_165216] (~791 tok/s) (#62) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 --------- * chore: clang-format configuration file based on LLVM (#63) * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. * Refactor core architecture and optimize CUDA features (#75) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- --------- --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor .dockerignore for improved clarity Removed unnecessary entries to streamline the build context. * Enhance .gitignore with additional exclusions Expanded .gitignore to include more file types and directories. * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update main.py * Delete quadtrix_training_report.png * Delete docker-compose.yml * Delete docker-compose.gpu.yml * Delete docker-compose.dev.yml * Delete benchmark_results.csv * Delete SECURITY.md * refactor: switch tokenizer from gpt2 to tiktoken o200k * Delete contributing.md * Delete CUDA/llmcpp directory * Delete CUDA directory --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Optimize training configuration and CI workflows (#87) * Optimization (#82) * Modify training configuration parameters Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * Refactor training configuration and optimize CI workflows (#81) * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * merge branch master of codeaddict (#77) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * merge branch master (#78) * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Eamon Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Optimize CI workflow and Docker configurations with refactoring (#72) (#76) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp * docs:report [run_20260530_165216](~791 tok/s) (#60) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs: report [run_20260530_165216] (~791 tok/s) (#62) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 --------- * chore: clang-format configuration file based on LLVM (#63) * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. * Refactor core architecture and optimize CUDA features (#75) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- --------- --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor .dockerignore for improved clarity Removed unnecessary entries to streamline the build context. * Enhance .gitignore with additional exclusions Expanded .gitignore to include more file types and directories. * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update main.py * Delete quadtrix_training_report.png * Delete docker-compose.yml * Delete docker-compose.gpu.yml * Delete docker-compose.dev.yml * Delete benchmark_results.csv * Delete SECURITY.md * refactor: switch tokenizer from gpt2 to tiktoken o200k * Delete contributing.md * Delete CUDA/llmcpp directory * Delete CUDA directory --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: implement Head and MultiHeadAttention modules in C++ * feat: add core Tensor class with SIMD-accelerated math operations * add inference sampler parameters and repetition penalty * Linear layer with binary serialization Added Linear struct mimicking torch.nn.Linear with conditional bias support. * FeedForward network module Added FeedForward struct mirroring the standard Transformer position-wise MLP block * LayerNorm module with serialization Added LayerNorm struct mirroring torch.nn.LayerNorm(n_embd). * gradient and activation tracking structures for backpropagation * single Transformer Block with Pre-LN architecture Added Block struct mirroring the standard Transformer layer composition. * character-level DataLoader with batch sampling Added DataLoader struct for character-level tokenization and text processing. * Embedding layer with token and position mapping Added Embedding struct mirroring torch.nn.Embedding. * BPE tokenizer and training pipeline in DataLoader Upgraded DataLoader fr… * Optimize training configuration and CI workflows (#88) * Optimization (#82) * Modify training configuration parameters Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * Refactor training configuration and optimize CI workflows (#81) * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * merge branch master of codeaddict (#77) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * merge branch master (#78) * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Eamon Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Optimize CI workflow and Docker configurations with refactoring (#72) (#76) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp * docs:report [run_20260530_165216](~791 tok/s) (#60) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs: report [run_20260530_165216] (~791 tok/s) (#62) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 --------- * chore: clang-format configuration file based on LLVM (#63) * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. * Refactor core architecture and optimize CUDA features (#75) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- --------- --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor .dockerignore for improved clarity Removed unnecessary entries to streamline the build context. * Enhance .gitignore with additional exclusions Expanded .gitignore to include more file types and directories. * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update main.py * Delete quadtrix_training_report.png * Delete docker-compose.yml * Delete docker-compose.gpu.yml * Delete docker-compose.dev.yml * Delete benchmark_results.csv * Delete SECURITY.md * refactor: switch tokenizer from gpt2 to tiktoken o200k * Delete contributing.md * Delete CUDA/llmcpp directory * Delete CUDA directory --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: implement Head and MultiHeadAttention modules in C++ * feat: add core Tensor class with SIMD-accelerated math operations * add inference sampler parameters and repetition penalty * Linear layer with binary serialization Added Linear struct mimicking torch.nn.Linear with conditional bias support. * FeedForward network module Added FeedForward struct mirroring the standard Transformer position-wise MLP block * LayerNorm module with serialization Added LayerNorm struct mirroring torch.nn.LayerNorm(n_embd). * gradient and activation tracking structures for backpropagation * single Transformer Block with Pre-LN architecture Added Block struct mirroring the standard Transformer layer composition. * character-level DataLoader with batch sampling Added DataLoader struct for character-level tokenization and text processing. * Embedding layer with token and position mapping Added Embedding struct mirroring torch.nn.Embedding. * BPE tokenizer and training pipeline in DataLoader Upgraded DataLoader fr… * Update model parameters for training configuration --------- Signed-off-by: dependabot[bot] Co-authored-by: Max Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- engine/main.py | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/engine/main.py b/engine/main.py index 78b5eb3..5c15109 100644 --- a/engine/main.py +++ b/engine/main.py @@ -63,17 +63,19 @@ def success(msg): log(f" ok {msg}") train_split = 0.9 seed = 1337 -batch_size = 16 -block_size = 32 -max_iters = 20000 -eval_interval = 100 -learning_rate = 1e-3 + device = 'cuda' if torch.cuda.is_available() else 'cpu' -eval_iters = 20 -n_embd = 64 -n_head = 4 -n_layer = 4 dropout = 0.1 +block_size = 256 +n_embd = 192 +n_head = 6 +n_layer = 6 +batch_size = 64 +max_iters = 5000 +eval_interval = 250 +learning_rate = 6e-4 +eval_iters = 200 +dropout = 0.1 torch.manual_seed(seed) From bf21fe93334e4cb018fadce78ff10c5c90f3986f Mon Sep 17 00:00:00 2001 From: Eamon Sippy Date: Tue, 16 Jun 2026 16:43:45 +0530 Subject: [PATCH 03/22] Revise README with new images and details Updated image links and improved project description. --- README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 7eb276d..ec07658 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,10 @@ # Quadtrix.cpp

-image +image image - - -

+ Quadtrix.cpp is an experimental local LLM project that combines training, inference, and a chat interface in a single repository across three execution paths: a C++ versions, a PyTorch backend, and a React frontend.The C++ version is a decoder-only transformer - no external tensor libraries, no automatic differentiation. It includes character-level tokenization, backpropagation, AdamW optimization, checkpointing, and autoregressive generationd. The PyTorch version offers faster training and inference with BPE tokenization via tiktoken, while the React frontend provides a web UI on top of python backend.You can train small character-level models on CPU or Colab, or move to BPE-based models with the PyTorch backend. All version are designed to be hackable and runnable on consumer hardware without requiring massive GPU clusters. ***technical notes***: [docs](https://eamon2009.github.io/LLMs/) From 394193ac945a562955fe30192be17bbd3af99c75 Mon Sep 17 00:00:00 2001 From: Eamon Sippy Date: Tue, 16 Jun 2026 16:46:41 +0530 Subject: [PATCH 04/22] Update README with new images and project details Added new images to the README and updated project description for clarity. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ec07658..eaaf5e4 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ # Quadtrix.cpp

-image +image + + image

From d7acb8d86726045d0d4bd9b547a5ca07852b0840 Mon Sep 17 00:00:00 2001 From: Eamon Sippy Date: Tue, 16 Jun 2026 16:50:25 +0530 Subject: [PATCH 05/22] Update hardware descriptions in README --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index eaaf5e4..59709db 100644 --- a/README.md +++ b/README.md @@ -17,10 +17,10 @@ Runs are ranked by validation loss. Lower is better. | # | Val Loss | Parameters | Time | Hardware | Description | |--:|----------|------------|----------|----------|------------------------------------------| -| 1 | **0.7176** | 10.82M | 61.3 min | Colab | Large-scale run, coherent paragraphs, strong convergence | -| 2 | 0.9250 | 1.99M | 6.1 min | T4 GPU | Optimised run, fast training, stable learning | -| 3 | 1.3145 | 0.82M | 39.4 min | CPU | Baseline, small data | -| 4 | 1.6371 | 0.82M | 76.2 min | CPU | Extended CPU training, 3000 iterations | +| 1 | **0.7176** | 10.82M | 61.3 min | NVIDIA T4 | Large-scale run, coherent paragraphs, strong convergence | +| 2 | 0.9250 | 1.99M | 6.1 min | NVIDIA T4 | Optimised run, fast training, stable learning | +| 3 | 1.3145 | 0.82M | 39.4 min | x64 CPU | Baseline, small data | +| 4 | 1.6371 | 0.82M | 76.2 min | x64 CPU | Extended CPU training, 3000 iterations | All runs: @Eamon2009, 2026. From 401f16cb177c6d7100383ddececa37797cdbb508 Mon Sep 17 00:00:00 2001 From: Eamon Date: Tue, 16 Jun 2026 17:28:33 +0530 Subject: [PATCH 06/22] training training run with best val loss of 2.9971 --- docs/quadtrix_training_report.png | Bin 0 -> 218171 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 docs/quadtrix_training_report.png diff --git a/docs/quadtrix_training_report.png b/docs/quadtrix_training_report.png new file mode 100644 index 0000000000000000000000000000000000000000..1add67d82d0bbe35f225b66f21e4a17743f35bb3 GIT binary patch literal 218171 zcmeFZXH-*L`!9+W8=|rik*0#6(v;q@P^1Kq-VvpRD!mg>5fKFe>4b>%j`R{h1?dnV zKxiRSLyr(b=s7dnz2EzP&!_w4-f{03hcWgh1YB8bt~sCQ*X9b>(om*5bM*`j4Go>j zV+Cy*nlk}3G{*!^odoYla>$#2Z&DtLh8{XDHXh!VZq_sqOOF?hE*_51XV<)}-Q1xr z&cXus?g~ioU9c;PN3DCqQ`zYuV7vlaYq;@Jz1a{9$%BX=4arc2cSj@Tjwv}ume z(5NUp(Dg}O!O|u&=wZICmwzye&mx;2=QGSb^UT2U^GmapFVAQ%Dz3aUvtW5=_7$U9 z`sL*rFWn3N4>BJUoDx=d(%cg|G|E)O>Qmfe4=8CoAK~iUv68~EwZgW&kaZs!cJLwp z^DDSgdq(!y|9l6$KK0?X=KuCqSM=}G|NYie$xCwo+rjvhfAIYG9|g}n-ah`{ZGa+e0bC`P<2Hb_cJ8YnapUPR|arTVYcZcK%7 zwEBDf8S7yCf5+z z7O=NkUFo)?bMdeHe1BcO+*#+jY_&E~yRnJZ?7FDTY8ewD?l^LvCO^*w*C?k_8Ox(- zWAx{OT2B3YtDd|VP~l0+Y}JbAH{n)h_s#4^$Rn=M+<)JlEUKm_y@k|`CY&k(r&lWDG781*Sl)?am|^}|h0&PPi_1w5-`m9fp|rPbG( zC_$q&uFu;ERX*#_m?hmZJ(dPLbbFUazGj&E&c&5kcg3qFbj^0e2F~jxgitE5CDQa} z=cm+Bc?04ulQNQfE9DKuky6V?qrP}Jgi#N+i#-;YP3%LR;uaS-sPiVQ2?t?{&9X{M z`)rX3=6-oUJUILo(p3|fI-z{nF3p0e%Z!X2E#VAVFv{UTrV2wxclRg#BIAzlv>bf6 zpgpJ1pX8Y+`A mh06?x)|z|_wuS5J-ON7zw$LK70mgKxde+hrYoXG7OaPaegy*f zR}c-Ebp>HcUGGxkRGR-qI6||1C$!t&!(hiV>G>M(sE1xEmc(e_%Bg$B)pm zK7B^0@>m@BJhLpfm|svFGuhyuLdw*DOWD-2y0u@ofi3Dn&wIm%4hn+E0%moI)&=(l z1BaR@hbD_iL6fR(-Je6g{Rh>JO@Vt`Hc87p$JwKJ^2eho{`l^s8DXab_QT~^0hd41 zSAm`JpZa!VfryeP@0c*0Kd3L(r)Bf_s8W1ptMPHaLJ^qDwCWmn1O&v)SRI7=+c;o+ zpe&TA90m8!Ak?*KcZpYW|95j>cDYr@C1M)A{C*!7=V)Ij@$E%U^-HlCpFiIhbD88D zuW}MrF^>MyQI_0Y3l5MnmwTDd@LH72##`}P5qWuw;@bJE2r?VEDQ=5d2}+J~_MDKr zleUP!*?K}Fk-5=+=niokoF(LF%FJ3lA>MVS^+f9M8#WOlR{!he*?^dbp1ADrHFP#aIQ&CZUts!L zHS6K&vGZ7i1-TxIr!5=0^K)|FZYzsRd(hsh!6M$iEN6}PaC?THa2HFke^#vVs6GLc zUmddGhN)RNY>_w8Zw1I$m{bz9r6Bg@G+na zIFM^Xw&^kN?315nVoxf+^5T?v`i*9BqP9HmXpUKn14#`s8~}SrcVk%|68goai$zFZ zP_H}8{a0^FN=jxQouuUmcv@$>m4h3HU&~S{nDQuD$=0K({E>%dtI>}b%+!RWxx?g1 z+cQyWDPm6fB7wTIEsyJ~hZQ6tW9ZisMOr(Zyd}p?tvr{9<7CMbXx)+_=Qhtkw5KA* zqUinhPbLeWw!q)tX(8gX{e~9%i@TCOzn?OTIVP&)4tln%QVt0e^=Ds;h+l@}u^nl7 zd3o7avm(ihIf&%r_|8i_&*#9sBC34Tr`Ajqu4#WmEi@mww^m=G;^Q(_kvAG;x3T;q z^lXnq9lb0*)!k|m>3dz?zyFJ(4IC|iHK}sEjgqGzjpWXo(N{{9>MMHk{IUpG|0##= zFCMCVSgPT$ySbR1|8*}r#G#}?Y(=$IZ!}WmqA`Qct0EUE#GzS@LDtxTkjy#;BbCW9gu?xpwL>kzGr64--2q7t&EoQzK6|5 zEAQ^FB0e*(ZBToGb(HMZ-8ii{?qd7t!un?j?zZ90z4VFIYMgV7(?LPj(hJJ$^P>Iy z@E6IO%9rQ{Q5}lweZ!dCp5U}1jlFI#a zEpL4Lxmu0NZFbJpFBjLVYiCHUR-iLM!Jsy06&ZE7)X<8pNW2swnQNJYjP)Atu^j=o zd?xmr-=r^I&j$x8xpNGcXVX%lAOMwxAAc`ocj=NLpb*u2kJ=ghwtyW}qJoCuzjuB& z)9-PSmLNxsqtPmvG6$Pz$PG%5IBtba8BYTB;%pk4 z3!MK~R|U0^xFgNU$@$ACei;MH3fia-9DD`^yLFMZNgGcW>4dB#Sz8^t3NBCnx5H zyB@08tHz3eMrDhzTph0#?}$h~m80ToY^}%e9!|1|^@3Yz&aL|(I|BN%Plv8KsT8Qx zL4S@I<}D#>3clonO53T(WHu|j0@lg7I{Nowkhd~ZuNl-gmX6WO{DP#dOg7YyntLXu zw3-#L)Gtag+%lrHFlcTvZ*}k>UZ*C2KHr_PlIHl+Q!G;}P%P9gSqN^AV|Y#|XTw+|A~`1r={dM&`xc z!s@N;OUB)S-w8U5ldhEZhEbce&p>Bx$ULvUhAc@}a_*DPopYGuQmc)lO~zIgc3BU2 zqVJ7x>fZdJpf#kLXY8pRi*c@Q` z+~K-lf1@ZxKWp$Hir+&%o$;Qi?|^*&lX|E{Q&0n{)N6I@#Kart29iEv{f^oNx!I48 z?wljyEEa=?7w?$EIH-^gfvJoslgE}UxOt-mMw=h}wETNY1h**sr3g(f|wb z>Z&UAR5*=`t+)b^sfV`qENxkSQ59mhp?TcPVB~#ees2WqC+ewdo+|PSgh2`=F>`Kb z2an)IR)Gt+3K9RS=nl)R9bIxx$fXaSe#WkY@;f3IC{t5G^eNN-w6@<_29~COpE!-~ z5BVLNxz4yFZ6=lWaVXqaPaUl6DFjMxIn&Zuda?0*{n;T;zuER^ zITio(sw1CH>)jkV32Nd!vm8)KyWFNinfzjdNR+S7+d4rFS`iL{JTo79{fuR#6!m8+ z-5>gw-d%Vg>$mxPXRoB`@Qg$n%)9-NK*ZyU{3Ng%Fw&3#XGDAtHtJdaW6E%%zKNnEhFN@`79b`e))98^T+!1zc$AnRykwuWGF{%okz$8)&(5y z;L8AJ+|ixz=<4hgQkj^t13jP}bBvz_a#!Hit>4VU0N)aj3w~=F56$X4px;zGr6!6% zojd*Y--{&ukM)A}cA2PoGT)uV8MwQ-&U`p(i_puJT8NL1esaB5Csqy{Zh!VNU#@mS zSF*w^VkNKfaCiALdI9w59d+*?dA)mI+t``hvP5A>|&{7yS#P4I&;Iy`sCQ#Ws1)KJ#V+(O57V-$P<0R6k5p-3AWrH6n#!fB! z;16HB|7Bga{tvMazB+0d#+GS|O1oYkcYbhax74yZ;Bc$tbKdEMa$D5a;Q?qsyIjXk zOv}?j-udruuir~8n7#Mn*S9k&i^;l+ zTTPS$)VsCbN(VBFbIo_#k?tg6$c=1HSy{7_bXRAEBlSSTlttKw#ad)DiGkh@J-IrQ7m&wM%Gc3bxKu*zAl$oMf1OG;;bK zZ+9uR{SEAl0C*8_ILyTfIa#+kpax-TF&_OL&XbnwA;pvC^F`AwPydtjX5XVZK)%Jz z;&JXM*(3F-^jXi_A{q0o>`6a`dnKBAp11%iUs6#&yW~Uc5$7#BwbM*5|5!sK5>;yE zHTqL#LT`6seXykrpgJutT%b2I=!8h{aW+2A&c&%-W6+Hu1B;{vq6;m1wpt*fzKv9enu%{<{-fKUmk%o&GkSAWPs1Q|QJ zQX%B)(+uh{+4Y_H2efRn!j$B~Ro%kB2Q!yQ}U zNzme1H_FviRlD(2eww{o;>Ad?NYcThn(pMSsa4qL$NtrXPxJR9bfKFQHwLjeSJOauR# zLF3fUq?ps#_I}_`MuTl+)926M(0MvJs`&IL8A@_6@*=v;R*FT;@wbS>FyE|*l=s>S z(Yq~@jn_;Al#4u%NjcCNf(e0JqqdKF-K~FwLq4}rR~)=`m$WPks5sI00EiEpjg20Q z+L~Y$y=-g}>nDA0V4VykV-t~NaF92V3L39$jk^OSu_$JmgnO0CmoOXqI~>$=GN!-o*K|BwNvy z9CZKM%VYidA;yv=`4y^p`k`vNOr|eF_{{1C_o`bv8(7DxoJ`W|e~&xYLu^oKZ0McB z9)L7unD0GPXm^?grSyX;SwIW|BQB{T!=ApRO*6U@vQ2 zaW${?@}Fz|nowgq_~)YevcttW0)B@q2JHZfC)NQ=Hl1JVHL0hZkFy&F<#lI1#YH_v z8*oU-K$I76Vfg3dYkx&j1qA1?N?uEdM7~LNagsP&G4=!~w|MUL3L&7IAr|X)sLBkO zS&3G;V6-|C?>$!+p9_hJyORQ)cx zLZ&;ucd{uWBI3`2M{2Q<)KHNvUkg!+xIR&?vgcs?{AH}^fZn^$PN*g`C|aB9`g7( znXku7Z1BML6Gp0CEF{Sj?sIVl=Fxfu`g$X42aAX6%Oy?mU4l)fW7RG$L1(2r7M^CW zgP+^M#An!SNG_Fu9R{GW`|SQJmf2YZjHk~+U%HL_Cg667#naCfeet#&(jXLy7ka3{p~?UI9tk#XKWseMGBIils4X<9*4Q8tvmU%UIgq8s#4PdR^ZP6J9(s*C ziO1jla{#SJx7nlGbkN~{e*Dk>ulZ?bNI3oH$3HI=X#PE=|NP?rOLF`FIjR4Q0GPx7 zzjI6ebMpUZPW?Z5`+`p~OL^pGW@c{7^hzxk-qBRXWf0&pzHS3Qxs#eq%DQ@b;;Nj+ zRY5t+tf`TJR*5UL0xwDZoRsI%Uqgz1Kr>E8C5bxd>&nxX{-;Le>eXBbwld;j5w+Ky z3am2YgM3=;sU4B=S}|1l%WqvBZzPxv%DTeX%M)j3ry~T#PMeL6j>g2sassI)PI|58 z5zwx?kCwCmWOCye?RgP;vo{^~OM`iCXoz?R(w8_b%d6k#^!&%`(*rr09MF-{bSrD? zj!5aXTM1@fNu${@KY=L4-<2Sk0O0+D0W^9>R@%{}a&UWnrbo;R;JCMKGvVYFntO_X z`hV_Zl6OSjWpLp&DEq=>n#)#5u- zCEdFTm$gz_OjB_p;Qm<4SO(kXmWFb6vO@9 z8T9b^kzIhU=xP$>c4irf6hMjW)+S39LutbFAIShTe3}j^a{CEGbON7|_M)~C=+`?< z{^T`RBc2>c%6zQhK9Ie`okk`*7egH(0VFb&bWlj#+LQ<^o0P|?I!DrU(i%PK*;N{v z`&=&ClAy>{Wigfn(vTHk zk#6Akow1s7(3pN*OUra5fN?6ZZ*y_{f`X`u2PD5+ze=3OtLCMA`4)C!!#IKr+5Hx- zO9Dmn)Jc0fszIF_(sa117AduK%cXg~ac?cQ^#e-^reditLa^}`po45(RM2qhhN!6C zdFhqAzQC1)Xxmz!M)*w~FTFX1X3nWHC|2D@-Frl}SjvcC9s#8Obhwh5^G z7jBQ;TmuCEX^boiU|^m8$s#~FxR2<%j2zR_h?Nu_)hY04Bn_LyBqc2dLLz?J-Bgz) zM=z+Vsm36<3Eb$!E`bpW=u$;bmO)iD0;og8CZy-RWq#(x{@ zr6E-v6@8)H?pU9h4-eYV=2wsQFW;m((VL5C&b$9!<{LBeAOD6N0<5%ZCtzuq@vcWP@PpNp=X(r z*ZM1^a(fGU5s^8#RlwTr64_Vk7l9S`t+1pHyWrMueyn03o!ds~)#SJA)u#uky>Mw- zHkbB~*KH2DA<2Jvr0Qr${(SGgTvXla>0fRJsS?i;XOtQr1tPFjcj7&6ue#zzH(4sx z5sVi#;=%5@lON$YR3WfqtZl6B?{_R*xM}UrQYYuVbzlmm2;ASi1|e8E^nVG3J_VLq z^>thhEs5$DKm#NRKyfH}&m!KdE}Up!?)%e&lzTdHw{dHAJcWwJoe{5wll#v0YBWF8 z43J=y+~4uG{Uzr^*8*L!uf8oRd$&a8RlXFDt^HN1;4`SoW)bq}mNSj39FyJy)7*3xeYV4RQKi7%t}-W%*Fb5-ee+?A`LBNKJSXjjr$vnJ z6$kmNLt2}jtD(;c4P>$9@$?PR%J*rN?y;VnV3?Ohz#NH2Q38t-2?$MBf z-wS>Vx^rW7o&_q7eO`@?hk$7w51;^SJXh8wAiP{-et+0HkLfTQl5+W{3kdE^&jGPS zgyQhyGkdH-C_BrtYc2a(3ea2Teav#wQ6!(~k>^o#$+t4wxcw%!0HsfCQIp+5x1fR4 z+Q%9}>xPmgw+A0NHcP@-B!JqY62&g7eiTs!^^kjM_S`i^*fuh$6$n|Gz;(}V$ZL}? zY23|6`51R-mBj7u6H)UuZd7XF6kUxAGC421sg-os@}+`yQ_z9cuWupq>8tP;3uQx9 zmHNqpGiGLoek+9-w`>i#eZ2KGFxz{PwRp`aCTue)9aD5;9SdA`>#^ff2*C79B#T`p zk+s0qVshJE%t^_wNMDffJ6G6NUKdnk`1K*X&#xnZg1=UH*&9|Rzd2H$-w5?Rb~qz?`FswN<^ z-8d)|n#N=iaPQZi-1e_yntpzDIyyS;!30ff`rhw>TbR;B4-z@SXrObXkY$N@6xr9QBj)Dh;&RzC z@NsrnZ6(<~2cM*YYWemj9avYDWz7kMt>Pf6X}D+G%PL&N z<~`9b?Fi(}NzWxvO+}jloUlC{-YNzwuRLvWF9Qm6Uz^coTWc#TAF33{K%3(vX+N0z zHlR8t<`N3%D9A3z3=pgRea17#hH*$=_itnYbL7Z)n&()zR1nbHdvH?lr(J}1yv4tn zi(%4;rhpwcO%Vwq^NLLHbKv%*fi#RC2M8s};b~q%Oxn=fD`4eKqLx5DDF*do5j!;iMD;innN)Ah&d;ArwZQu*MWHo8vIIp<5tS^;ERZg0+?KLQE2 zb5ftRjyqLjWvCG=@9jN86G4`Hm_VSw;ht*a)>&}clX#PM}y#fUn(18{;{;Rms-BUM=P_hQ$^Z~;k&i?GzU3b(02<4 z_HM-eX6_B~^<>c3lJz9; zbb`B<2h;@V5`CBow%(XO?L+l(X@~8#{^Tt|4xCGo_ICa@``xD~_Uq$q(+az2@lhRl z7ofgOVnof-s$AijS`>Gh z( zAh<)mcRhXb@hl8(2aLMO=gjp?P<)3S`nuK_`uI}Oa- zmrokLvOEOOru%JaAPOd2!WQuR`-9jU763t=A-FpS3^>?} zdX5^|##PFjiqk2*oOH}Z56zICg1KF}>ppf5B;h=IZ0?sFsd8Fgw;MOa6jVCM0#GH2 z4gy{GPGG(JF|(I>BV>14mOwo0@n=2f3znT&9V*1SNZTA@ql>9(DfhFgYQX zYJi??z}R^4V+|F>UeGg+w5YcpEM~0OySpmz0P6ig9P1H#`)gWC3NK1-&oPvIxZcJw z4_kfrK#e3_(Y;Si+fgqilx&CbGd4F@>!Rn7o4|{%R;_%A5D+peRr~=WRt-0oW-NP> zMK^%ynBuI;T4!46aK<(bBp|LSv%0?i6d&IS#8ExVs~QGp4zV}ws!svL4`1Avp$1Yi zABCNt#8jjNQ>T&)45U{$FHu3#{V0Ocj#Wsr;% zqu>B&F9U<*6Xv}-rpN9(dy%*gf*xX6?>gwX!Sg|o$J_b!ivCVvH8q-s+@6kz3p%9A z)Cs@Gdf;mb6(+R`fBWG>I(5$Za)G;#L2nFr!*5vnEhl2|1B-aF!0pP_60_S<9wt|S z*er>;>pIeCIsWcR3p5t0gKdqRMunVp@pw3ofldwf)lU4S>q3O9-*aW2DNn06MU9r7MKY|^< zS7I_0my%$6h;plZ0(b5@s7?Xqw!TkOs{@E9x{X0|I!;cBqeRGs)4xVG_AQeyj>jk>7Yc)tMHJl z`LNEs0yD?UwoDsZ158C4|(M=lmHa8$7+I=r&){B)bUR!7tPxT6-_ z=L^a|-_Na8mSZTKYLXI^G&?JR!fVtuqdmMW@=#VML+;@&~2OJ$@ za-U&yg6H#0;8c-ROf7#FlP>8?4~QYyjNKNWc=3u{3Ebm>)OH-Ogpc+E4~SdV+!>*$8A|ST{-cQ(5*leQ=a9 zrFyT`qm9ZXnhSKw$r5;q(OB^*ujT~yE@=d#YRxKp!;yKA(b;6j&`!fRwF~Lkqz3tv z8=Qez$Ri1y*`7*Pr?v&k_e&;D6lVZRg($T3NBW^ri&AB7bDd|FW72HWm9W2&7 z3+k4fnk4hTxiH(;H;+1Ra(CSiw;RjGY&A?e8p7uYuw!5R6WMXg5Yv!iqTW*Do`Qxn z#AooM0^@5JQ zbRF*SAw+~2rUUux+q*Z3=A}e$suV~mshK{rI8=~Oy9^Ob47LC4P|4`@YdQRw)3+rv zWpP|UdF|OJEcE>n>ledYpNi0TH_+vS1T0ititdh?#@d;*Lmu1I`l^X2;Cy26v+9r; zYHZ6+C`SK-4hTnwdxuD%;AcLE2xvg9&llav6&L6Mrk3tjUzm<0_eISuZg-Q;oPoA2 zb6VD3T<`MXv!Tfs0w1BzvUU1(3i6j82_`}x+LoR{bku3syUhWr*F&`9KF-_lxM=f7 zp?Tp}4pV{XZHYdy&w3#aLQx>FF4mFO`yuW7r5=XxJ!uKlF*^54;v&PnbyAi;UI*h( zh%6fP57W|ItF?1WMK>p=u+uL0810_Y-3b8jcOE9I;T0Kpz!4f6`XDOw7#85jwG%&f z_#j&6wu}I@)9gdJxf>MH0|T)!8t9BEOVP$X7E$7rA2NxVQNZvYeh>Bw+*r>PR|o|7 zkSfe2&(woaL|3G2`bD;`=oO752IG2eH%^_csD3jtz zS#!+0^ap;V?fH@>1s(MWHPz1HjxdhPAM$OnQaDd-@@c><*DvQ5W&ZSV{Hyn{_=nUE z2{Zpj-4<~d{jmf;m>X!NQx(5)4lWpeG1M?IhA?_OSDWN*-AJftHi1@-=(%!fx_UYY*H?ywXZ zOifyF2}rG;D+~U9uO= z)4^*p(33}=(-Y`m$;Yqe{;>U06uVYDz;>01qF~B%x^K!O?(!-LhyCDFGaf9A7=pKf z%xqHe4PhanS%9uZWv!3bNu)iV7&3(8L*6WkEy(eFu6SvGYrXC?FtruJ*opSizvvCz z^lLiasK%ZJzTj}Ag7D2Z+I(cs-7mB%o zjYE@jC9tie_Lb2aK%f>gt!I?qf6;JO$Dpdrwc=SOxBgO3(AZ?3tLP#T>)q4k??_+| z>R1|zsrD?eLiiU?{>+~Ht+<4t-^Ef{d#qv3y?!Si2(P_;=!9+B#oGe$S>rBc8|lO^ zGfsp>PVY&XJVG*49Ns)gsw9~!PrTbKsP?s@OFp{lL4traW%*uHRgaWsO#Je2$Gg?l z>=z3?seXYQA8@dQv!)I7OI7&TD|tgQ21POWYTT@({cL-gr`w+7(NS}wL0kAlP)1>b zmwZW=T#HbY2r$br{R73b&7(LyL!N`)Bcr97r?`N0JTB%0!ox^g+PA0gfQ?3lc5hEO zz7B7v5LG&Tc@wR1885*i5jp!qC>ZK@K^&#m8$`@ zIsGksTg$KQxrFy+$(Nq~c)442%6fGj#F8|IiU*9pX5YJG^h?9gA58wKk_qm9@AI+B z(1)qqLr~G5?6Q8kxvCrs)1UP9Ja`_D`Q50is?PdU&p%zEv-vPw|LmZ|8kqGDa4Zp9 zZ|Z;Ka3i66ngz29iu4V?rZl!a47SxPJcYDGxgeQ9^y9g+s?DFAYJE9Bkkygp#e8`k zairecC3g062VRQ9&s7_N;a)Rb_R=5q1!V1`OsY9kg^5?u(&b`muxQ_6NTGk}h1@dW z;v{{#58MZ=o`9X72rzSbR`4F3<&*1-=2RIs8H;QLC==qZronPnDXfzzRdpqZCYEa#NK5FyG}39Hy ze80XSC*)OXN)UXuW7T!bh{p_P)GOb_XGG43;&Lx(=>GiWF8(X`(aNY)c9SpH$Hr4o zQp~^J;9hKY68N9y`arG`E}P_I@Z%Sjdgqr>~qj8(~p9psu<@&^`qB% zV3e0T8~5t2fh=&AESV~E1>o{UYk3Ibd;N#+g>shm0iEMTW@Ah%oV1wnCzYHYwSY}o zlH!9q776cKyxp_YF`Zv5sS2v}LK#}Y3Z^0L2?%Zc<|Yl>#L``|XKBc>N=yC2>aE5X zD_^Y{xk>4Mq)R)CIUBAvJp*eOJ1<=mz4xRIqcE_l(oe@Cav_J&9}~u*y7>C^SZ?uV z{SF@tqA`fVnOiBkJoUe$RT&6Rn291E?o{bldh|rxabSyB0rzGOXEd%k1m$&jnX{Ev zD!b!C87}IfOqnwciT(OJT=EpgD~}sLBtyx|yLsa5jl@VRm!=(;O7!```~sl1$GF)x ztSjwEWF)*kWv*^l3y0m@se7x>Q#3&n@7Z znv4#|=rXwD^oHU7eB&%s`5;c%Lw4}EHqVp}5MRybAI{|K6$Iq_xBA21|Ew!qRV4um zdeZ4wSpN2vH5oDJK#OS$eY_nAzHy{TICyFnwKLnX15i_S>`ht-|0Q11}jV{^ovWWQrNpH-Kd-_XSd9qcBt ze5UT2;^v4U-Icr7rg2VS zKavn-xss>Bl6Rou84l1^+cbx$^K0e|fVA5T^WwRSg7^Kq(8rZ1KX{E*beal?J_Oll z>#k07bju@Ip8m}6$t=V8#O-!Y>k`S+Uv9XCO|~S*+!*^#UQ95(v204r>+lV6p7)4v zn%v(`^5QSKDlWEqWwlli1@o06mNj%{?^@Vo@2K0F+Z^oVI3lELwF=y&aJ7y~Tznco zVNNDEPXnVeJN@{)TuHkexcalAVuPtk$A!&@+X3cjwJJI|Ld#$u7pgH(F;Yr(3;iy$ zPGizdZB#06-G3Llv)INlSID6G7vXANvncoXs_Jg_o$OMr#xyyz7!@!VgO_Kw+*xhl z@7*54JOe>AZk_@R7MaQNqb-rNG{JY_z5tvmw(N5h@9zg_?rp91>`}R$rnz|@?EG+Z$=g-Nh+S=N*i=0Y{WKR&Y{DFrcPxlAK$5zf3eEgBN zq3z9O4C*~q5Ar`eAX^|X28D)GzQ00fWo%Pjl@z9_Tt@$L95-)z;*91E8_PTABeF&&~4~uQcI+?cYh0$l;sA&vn@buRp*ZQvO% z_G?#U!#)#CK!UX$y6kiQ;2gzM3njQv%Ho0qCIv7^(ms2cG9RG$@8bZ0#;ZF$E7rOaUi%2=a5wza=1U1);9ye6F09&Uci&I+McXA zNP%cd26GcgYo>vG3AUUn{5E-mD5bml)!xeTzZqK&#>PX=(`eyQ4#BJAW`d6eS{aai zq|-U_ix1J`fNCIdY6(<>Vo}z4xQ!|%p(Ec7dZU*W>mnC?fGtR31<{2xumiAHM!&?I zEt-&xNMA-o7A#v6CL0=H%QN7bi|$%nH7VyCKnGgXEZm5)EonfuQN-;#yLk``9j9<% z=#v2}_o8^D$7gQex`k*xE4ZMV`GEo7+>Mi;a1F~*EVr>HCMDa)q^*HC9nsbJs4HHMSoXW52}xicD9_nxQU-I7it42IyBRe3 zC?7Evf+;wsaT&{gPa)Bv8Q;~)4a>T<1-C?z4q=Z4Eh6(F?&cNQ9JJ4#P+uF+;n=4$ zssV$Pa}M<^3-m`>;Z{Q9aesow_nV6;{Ywn759~O6-6`};S7^(TkAM+wXo4?cry^i}YD02L* zhh=Ky^cRJ{<~OMlfjK&?=|dNOW%{-KT<$LG4R0w{}oAa7%ohb z{5*Iv(*4;(&4=4O3B!ZHNI?>_x@@RB=!gZhjZ!O#&m;4GvqMxpQ90~)mrq{+Zpi3& z2Tuk$$g6zUnbNEO!@7tZ7|oTNha;FBU{@+;~0vCS1dio4I*_VRz zs7v0=Ue4OHk+qfER8mSAR$)`dCB())A9L9y8`;9=!nqpk)=ZxcR8>`lh#vXIoXwQX zxsKgbYRQO_8>=q_h02y_1dPskmk$l*Bd4v_r+?Z6;2Xx$hR#j`*+&ev?C}G#tqa0T zb>i97Y$}iWX~ZE@XDW*pX(G(O#@w+0`Xgr-7}^agh+V8_?V@chx&JEIxbQJGh;rbK z0zQpP@#zKEFm~@{5*%<&*#+0c%LjeWF8Zl|X}h>9Cwsa7xkH1bY83iL(hAtZ$4WG3 z)09^$30b&Tb9s6j?Am)ZDPtJ-S~DwLSE>g%&`C2l8ah;CnMfyZ$-X&*zhV)~@6xzq zBV3~KC10)r&aiM*%bK3@T)d%QWt<1VyC{$k5}P+PdoBhx3Za1E)kA08zw3N8EF)5? z)0E%_6Q|6XUH*a%%U!COk)jT^M0Qj!U5B!#|QO(m3Z=*SO_xH9HZ*x4sUh~?OiR=&f zpo^O81tsyYw^jXh9_zX)O+oBY_5700l}5eyXXsJX=M2c1;L&|4EBjSWFitu#O;^edOHhw6Qfw-3J3YIueDr35BA97;)o^x%2ipn*~dEYvOA{{s{Z`_7!t8kIm!o+ z8vi3bvhZZC04YX|%9xzAnZ7Fvj}kJ>f1_9Xn>MJ=B6&(DJ3rL&?bZ1WAGz&sGt9DM zGb$>#3w|wLk)ND_PK?Ze?e=A=h~!(oy^`!v?sO2B?E$iBz*m(8k&alWvEz_!5s@RL z;A5RJYOlE1Xgvm0*rT|;8~4lFi2KCkRedot{C3bDk`HWd-I$f6`1U)5?Rt~ftiubV z0{thusRb*^KAx-6jcZUqk?6Xvl_TR7I z1<7T;XxQP!QCp5^l~@HgHwEW9mDzn_S6jbF(jI}ycjaF6&j>~1iVLIO7SRqRjNo}9 zj;TC|ziLVP(UlI0C+RLp64j%hM=>VJTVy&IL@P!pNO@mQk~0HNf9GUI$!eqjs7GvYM9B*bROIj^>MYZ{3TJ+>4e|z%y~-Su9(z?iZVI{qQknzhwmUVu zwbfYeGJCyIH+E_L*$MY93&g!gRueU4hA!jWWzFYrJa2uQ>TN#MD0rzCNWizL_LnKx zy5j~ExE6n};KUpM{q&lSgx`6mL&Fyi$cmA2b$k~qo^TR=0Whdy3kl#`<`^f-YW%Q$ z@??lXreFcY&puLDi&cF^MMZAVieq%KnWDLG*0cmVCt{__WYiX< z1KW|l{d?v#!EgV4zRf>pRVh40oKKlS{7y>X5pqs#-y=j!_!YD-mqdVjva@Vb^_JB) z6F8bTbaLJScs{xv%aahLqkTij>hWmBWRS^hX6)at%ht+<>*0Zc)k0rc%aen`%lw_$ z1!!6tuGfD>QA5V+|1}8rMBTI7LuljmT2}@;(n3qZQ}Jmg@o)3uBsgHEvm0HC`h^!j zoY^t;dE`YSWDW8mo|fa&%>V(zYXi0Jqosjc7ooUD7^Oyc%v*WEIj;QW7|4)M>2$8=>xz4IWL$KTB}`y6#;Dn%erW zsqz$4u_%p=rAX;`9{>FBuN39>3in;u@O_WEVr1hqG;?OyX=$E{Qst?Czgh+Xx3>aM z4Hy8s&GNvus1+bUQX|jd3i#J4;x6%E>z+n0*nvik>)ZsvS@2w&SukJE?Xm~Tjy&gp zygow1bs6G_T(}WE)vTBeVW^!sA5F&+z;L~3M~NEfJq%hdDr+K5-Qs7IT)Guap5!eE z-dQTJW)$^IJ}j=+m<^#-0P5;&UXB9bMN7oOrjXFk8%_J#!$o_aA6SQH=jT4=;%xTb zm+}NvhL~AVW(%Za6kMU7&Fhw$mgk{Y)QrbW@yx;tn)?qwtC3GZhw|fqucJ!az)RiU zbHAtpJdY*Mpd<~*2SR|#p6%24`+ezqrC>Qe#x(}^Y>a|-m<96ALeE>zfyeJ;n1UGn zhL&fHm8nf{>TOyn#_=j1cl#TQfOB0G@zrVT7drsp)M82$-sDhRo88JQvZ_Gy7G91i zjw22i$2EA5-JbP1aC5ibnEol0>tJA1eEO5Xp5y3^C<+;oI{GzZ|MQS+j+jru+J4w?;oX^ zMr){-%;SGVar_z`vXgsY4n4}Rr!gZg?bJOpG5dQE3#NI>JI8aW;D)cDlq z8rB2^-EV=geAx z-3^jTN_UsEba(yM#(SUZ^Ss|bUI#k3;|}GVefD0Tx#pU4F5OEve=sU4O6T&}v^UkQ zv>qL-@$61c))nQlhQJLQtCuHnRWaP>uCk1bR_(t14i9a^UL;5QH0R;WT#v{TjQh2- z+l|lTSq=Ugd8sWD^!DUe^QN0$jjAsYjl_G|EPOe8Npl^<=P>1174@TPw;N#Ncy8y? zjMq~#SN>S(C(C`g{Mh)jhX4*I;$wqjTkK1zD~uJfu;z_W(xWVoYyn_7?jy-nEAeYt zpo>HFg_ks|NSU&5Rx)v!vf%6YHy&auzxr|#LR`VzPM^MZf~&V(JXiPD8wUm6Z@VWp zNq=sgx$PuCxKQ;W-=I$&j~1+iV!qm83+{BY@KE}J&%2r8EyBiM^{7MwlJL;9Da(CX z8iIo{y94bOqlP?1(gI1@kfS=_)!3sj}5 z#d;1S{uyPyp6@RE#u%3qKqol7AZkzk@hO~k9#o>3%x0q9N5T4KrrD9INt z#9GgGu4*k$h50(pCP^0>MF@$1)P!@I@~IkioJNhmQ;P^=fIq3@Cv_{U&4A~({=R13 zwK4jX_YG0+ooG+s)dK}L#)GrWFFOB-dmjG$&o_^_vGT8G;+`qb`g!A`<>(FKRMPWP z5RmkMz!1WvaHBLOET9mD9wkUbHJ|i6vDvkGm&)=ki94xZI}XFAv9D%vI(2x|Jkz%^ z#cSf2q({0+$8GW>ptMkH_=bhIVr@7^1^?%-0OFTtg!kQexuU-9Pi7<8#a*15cm1>ovF1GSZ0>9}-)ije*J`kTHV}E9dLUoUS{V$yYWbqvX-e*3!ijR-9?JbGP9b)%tG{BtNEcHV zJbJ!0z_#9-H?Xub`rQ)F#+-+s6XRT$U*~ire#@w+r^5_E@~@K)LOjfI_fbm`hL%7yP^HSmQ{mv#^fDl-Q-j?rP`Xl{=7zMBzpzX zvxG3)ql0);t(k}C+!GufI_?*vxRixA2Rj~om)>d!3Grn!iSbvlj>bf;&lhU+X%jmZ z&WQ6I{2&*5@GIU9lG_5nM-w-ZUR(tv((n$Zc=RlF8%n=Nc-BuMG=04Cz$j^{6qVG% zO&(WELL0fcK&kcpB&Phpl6s*-Pfpp1yeuYCk>X`DYJF1 zpvyNkr>GB7b7HrP<|r0C7)8nSRvJTwZK%;(DoGGjVu> zIv@(JXa5<OF$tasc)2ATm#TzjaS!0Y8i;8T39g&khybgdjhN0B%+HT z|H%Fb@tnI+d=3PQzMq<$AC*|6l_jo-h=?CFbq(;_p(vu$K_6laAnIe#?jZ*t2T;Ch z1ok|6YE=Qus6S|(8o;vEsLe?j1k|oRO~@>>JZBonhk|PXl7O|Xm&*%s9_qmBrimy* zUvP>=|WyPv90vt^CzA&Pw4W96H9s zgLHyhy9rAzKYrY6Xh(7fST1>a*VIL?#dWkz&bDsNmKZ|929#(#=1oos;o6QB;vxfc zqB9>HGB+qB=ijBFif#)FizX|@WoAO z&Rcvvtt4@>S}jQM^M*hH&JcQVgg+?Sj>kluREMXn=VUdy)ix$-=40uimAM11Hjn!x z`cYrDcdLm7$JZc}M2D&x@rc;r0b`Mf9yLGzibpa8EJt}gnl47#13?M+dt%V_Y%CT_ zFh@GbOACu=%>~jAI|$T&@fP;8TM7sV>Z7wMEG|}+tFLIOEM}xX#~>t^f#G^u#@DGB zkJy=CZN8ylv|mwCbQTPytosfE1xC{ID?*`Ey6MjtDVstuqjz0}>J1C(n0t!))er5H zwtkTBYS5ekgWw#MMBv*RuQ(GmwLY|Z7Y)X4T}l`2SfWW{mm2#EJ=P|uB{q;hyfm1o zC4zAnzqH~>L_Px^Ef=mOy17W}N1*{V;^ql7B5m)O)a`R#R zadbuZ`1yz}1!6(=Iqcf2c;=+hbN)5MrIA${q|tk_4rxWw+;XL=6?jNYBiPP+23L+9 zYsLHP-)c(NUhPfzNlXCtU>R?Tri40aC__}JnuKWYg(xt)gzG9Ky<#ZQGo2o* z_NXM`^_h7e+xI;*Eqs3DzSI@`;w!j&SMcGh_=@*8)WAkF^%k*1)<^eYo_(pJO^r3FD+Y~4IiGv*12#2YdxUh(@i zAC!{qSX`SnZyjIMk+xVU7jjOy_7c^gNs6r}QOXStr?8`UCGvhYnH-UC#;(p8zB6eE z^0vE$>9n8;skx*b70|eM{+7nlinZuQXL5LoTlapqtfJWV>CRF+tFN=kDHHIqtx%WW zYy{WAgwxSp6q1~}ie%ztn+elYl2lbOR8^v@s^AO0*&>USw{hEgToUTvC#YV--@~E0 zH8{lmjUgBR-MH21=uF4g8>hOe4^QQ%lIG{)OYQY{=HjOa7N3+`WVkof&^bdUOY6}8 zS-&TJ%E{`fXIl&=Q|a52;w{gY+1lBL9(B&ay?M{3tb2a{w7>t1!}v-BWeToTy5c_Y ze#HYh2E)@36(s0Cg+vy&#SdZt0I0!kF;nhv3U} ztDpY-^7wA;6leR->oNwM!E^+}u_XdEvm#al@S?$?ENcLfT}o?C;^N|+ydt!AvxlN7 z*JihR4wUrt%xAYK8uEih-7rm=&z_V(GU5AcktS!MjjR%_pt2`pz4`Jtrh-3ze=~{V ziTFTodsRY0s8%1nu33zB6WgyQBjGe$xU2``qnE^;EfkwV6sZ}{53u%dm-pUk@*HNy zMY#3%#K%V{I3Rf(eezC-NyA7mpraI8!FN^H!A|(#XlrqZYj)th{I>a~HiY6Ww1~Vn zWD0@XcC*3Y)_-0G0K82M#&>+!d7+&+`lC7GUfSSUR1h~@#AdCJtSLWs_)Q7z2b z0wA2HmAibo0_Rkj)#^mu%b~Vfh%cg)2B*+{V+*MQ?MRy^YK}WvenQs`B4e_6JsuYf z*X7bv;i#+kIEJiI|Ie$AvFD_*r@G5y-2`55uuf4{3x{5Gb60h9b#QZsFZd>csm%jV zmM`Z!mr7GzGyD}B@dkT&n7{- z%huC>xK)0Hft)U18fya&Ihj`*by%I^oz0QVG0>If~Jo z56~lBeYFkNasbE-1dG?&2`Iu}_}qR|_eu>ZOO#mcq`KF-KYQF^1!H%;{cx24Vezp86xU(nI5tVY)Rb# zOihQ*AQ-=`tO4~=piqK=jL#N%q&{CtMY3z2#HGLb2>#U2VqBPJzd(1WTC#x*Ztx({ zTnP!U@diR3V`7qJbrT64hVf<{V6oCl^H9zO2(lHClJc=CZ03LX@J*_6K{J==wA{4f ztx6Eg4v+2*t(QU#+qm3u2|@HPgT-sxAc=wbVEt{{?x z4be9C=MES?UZ+{Lkrc?~g`@4f2K`lTij-g{0-Vmztzj{ZXLSEPc!wmMAAiMU#bcOxupW=o0~ z+u4V<3(Ma*YDc_JKWa98Rnyfo*l~$1ojlwmg<180zE$wEi3Rr-+btcHxH-+#noIeE z5jVaD*|FBjjaChZ7=6C6<2^@8?!eyBGyP`sDyI=$kt+Wf6vf2w5t>_DOP4>M7&p6^ z$hx^Wj`i`x0>|0h=!nFq@8}rr&AifxM00LciSUiehg?~fi{x+9P#^m{#XY+V>-xnz zb*qcSFkEd6MHzOGd6lvri9}L!S6ns1%V-yl63>HyqOWSb1h0F749f7LKW;+{p~TCV zD$_VvS0b-T$SHAM@9?{2oBe(0Gww^MZ>I2%G$8!}nA$%mKo+Jy%q}$smtwnIBY$+@ zYjb+m(?h%(l@ad^tyEAQg@*$|+|vMEYHfL>Y_rHDMb+=)R+Y_`Q|8k( zXp%gT_&5o zIxf}lq^o^KI^)s|BwQ;SG4JRXeIJwR;})PvV4+S(^KZTiU(e?F>ck@z}YpcCebMzb4a5cMJ6^~*>|0GkG>#a zv14%W&QQYF7JAB1ohpHrjUn5)H1q>ysvIFB!JD`|!(HPc0z*%HFEP1)s(4hV(=M=7 z-*jU41tOcWMQ}BCKb4PxUe5zga zoqb&G)%P=|NOlvXPEcD7Sv>s~{GMf~e(Y+)2JU$)2agRjS8GU1?;0-pN=FYZSy{_r86Y~{zj z)SHgO=EY~o9u{PmXrhZEYvUMDk69f#*GJR)wM^3!lWp=*`#G*U-U*9Yp}I<3k^Z*Xr)?^#~GF4nyXFMRj&1-H8IRe46<*?BRCYdYjRH!-n; zYFO5)Y&zO4xQx5iEGys|x!z{vBXj|d*ne3(7Ao7-`%2{t;Pgx921JI33p3TNnG4LJ zXD{Bk14u0hafi)XW4RBDmmHSQO%4)F*S9$l)1wX?@D>ENf3lzY4_K^J^U-td(zMMT z#p0r<%)c??pp@%um(F;T$$b*uR^^JPgBxbTKhh%bT1)WV_C%l9D2M!5^zv)BMcQN- z zRip#z3vL*)$hn*|wZ-uFuc}2t?)2jkd-6DNuuxM{`bsMS3J~GOyD{K~;1zC+fAQlA z?YDMLm3z#VD++#tMD=5Dt6y>xldecfOIO4cod5UoCK5BVur!S${P7Ss2k2@*-hj(~ zh~yF$NP;&!5ZHkqpwp*-gN7v|<5{^H1bm>tbl?NrpfEVyCP2>+WL%bHP2LL80rP7% zZ+>zA(g%feY51!O?3batHsGyG6&N6O+P@3q{oEZ}G-*#)<Iz6<)n{zd31O%nF0)}!YI-YP*f-t1de&5eSQ7JY4+@fILCZ`{;mQtz7$`T zXM8_THAm1wc88S9=U)#aHNY=675m1FZ~Jf$(sw{L!*n$FJ)5PBgjFJx6@!6MKgL?PP6j*tlhW zV}=C-W+xTtRSG4kBqHe&!c9M@fW1F{FCGAHO*mf!k6k{;o4G<@sx{}j)0i!0W&QYW z)^kjUA8%6_gkZC&s$~dJxdW*utQfzxOvp_|(qBXgacr3tc(#AOW7cWCgP7Ku++V~I zTw%Q`dqjF)&mZR0AjqGD7TOOq=fhm3uWI?nq9cU3u3}-`3DHq+fW%78bS+m zmTCDqv$3OW^c75iN|f*{XWUbh8@GdG%#d2})iHq4-1)1`PY9m*1Sug;bq}#lOC@~t z=RPX2hC8sGXG@dm1m8&^F!X#|f87#grvMOYYX?9tjc0~?l=!N4Oh9(eYBp}F^__W5 z3D=KDKkLuMIf*-&sUu-=o3F%QgC{*<74FtE+;?YL)#dpsuTsKAZb!pKC+YlMV*qKtzppR9FI?Z|XYhz*h70*5}5yHI4g zp!dEhig!e|p#;(ZO2*~l4hV4s?(n~j$SV$H)?|h|fK+}g# z@4jpdzI&VyZj&#-(euD$Ou(=3s7;U*gD6Mur41(6MH$I?Ex~)CMU03x>DX~frR$*Q z^o38mTzKu=o!oY*FA4%!pYGtXjjkbV3dVc1z1Vr(hi+=3IVtznmgG=zOH%}`gXyk- zec6>@svbS7=szp=`H>{tCjYnUT*52b@d$JtM%k|r{2k_!26kdN2plU3WOyQ~{fa~dlK z44ZaJ35MiB&hsY79{u z|GzVqiiEnzGmqmexz+uxi;%?)4r#f$or8+ype_wI$^l-N9J+}6a*JzgYbggIfZ5k~ zlqG)XSp@1R!gd972*Rj%MGS-~bB>oz+&f~+-tXpCajFtCTb(#(Y!i;4SFs;XAfTuq z!WCi#mV)@U_TO|0c8=qxx+7nWqp=ocz58~@0c#(Cvea*`qhs& zWshe+A$ptvwod4Etz5jQj(2MyKk&yGZ$1WN^2;L1y4N)r!3(v10_6)7_Ed>GJ8R>+ zjz|$x;Xx4(3&@8eO9fCqDWYThk3(CZnbUV{^|x-0HM$^DnN)6 z#VCsMg>}W?&#S^~W4n$}o^Dmg;8?_=NH!R@pktFc zIE|~Rnz5A@DyTmpa{ymT5t@rg_U8;bGbK9+f0w(B}JpsX%+zqhz(mC|vTl1eNu2Sxi=ia6oy3U_%x;$AKyRGi>H zdU0P$xp_mEFzIl5`J~0AXI)Ft*ach*xqe&5X%{2thO$oP*9|GqDR&1gh&*M8VLyr0 zTVK}BeO|;;&r)Rs7f9S+Fs%i3Q&0Ccg@uw*Xl%mX@c0IdJp|_8s>SL&5Y>ZCMi7is z9|lY^D**>FlJ$n!?(De@-D&#rP`I{!S`gH@5ym@Bzx zp}fBTh1}!f%Y51M9ogC06TpaW+Wg84O{AaRHCLSB;Ie8@pt9;-w<`f7_wY2#BYdIv z8K&?&>BepKU@`7HPWCCRG0&6`>||y62a3{#u}_u+4Z1R(MLjmU3M7lVI(z}!5~X8l z5sNC#1Jfi!kr0xCCKLA@MAm7qR*RYW-(U2>#;5uL%(DTFVsqF73Aj6XI(U2s!2GRF z(rmT?+&9+V*k(z zI85{-M@$>&yNr%p1}=NNEMD{#VPG1E)O2$6LQ!&V58T_EQ6s_=4ociD8!pLb_4dA^ zd5pK<7HR;?`peA)m^u1V%f!p>*|A63uVN7HC24jCA#-7(qZpX=?HEeW&qEKvBvLrhE8r6`c3Pu+lz6Hb*go zj20rSo=yNfeK4dv zC1^SVr-G+#MS?B(hbP|RFx*>Hk(9gxF}~hgu_bC);Y=FuZ59B*vR0nGeETmLU=(Z# zWY%cFQxWv;GN;fmtYK1wi508SV6UeY1TgNf{;VMW-{P4{nkDhE2&|u_Ehz(sI0eMl zn=FSw;!lvNij<2wGe_FCE!`E)o&-$ccVDz2Z%;x;cMA!`5)VSwEfQGR0B!=E&8O- z!cCKlse3O+v;Vr^W9AwGY1I@ zVf>V)wN%?A)0#^QZh|LY+(L94pg|Z+g8|A!qu|^2&)!BnO42De9uYgqrbwAQGtnnE z{QRBd-xptr9(Bly52Faoz* z;Gn6>6tU_t8}niR8{CHjxHq4P{08X!PET&>6QgIA#s0#&3U!g3Wo1yfKml%lOzcAq zvxXftVYlH?%?99I53?i`K1$plxlsV6JeGWY`ltA6YO0787T^A-<&d9eACSGa4LE1&pu~ zd8Y8+5fkp7rlw|<-SwZ3xL>y}R4Fz%`M0l=VW`u|&*jcgSB75v5^TwbLSLz zB~|BnuUs%>|8^NzXUuvqH!PvL8rBWGv@yv_)jtn74~b8^w0CTL{1xxh6~CYI{NGQx zuMh^~gXVB()M(-|7FKR~#A4s&E)#1ao?zWVB0vFqx1csFT>rB(-h>^BgOVCo$4;C! zM$KM&olwg8n9XiBY!pA8TR~b{Mp1Kl?2mr_RCMe6Q}f^Qb&zuWQ5*Cr>OsphvO5<8 zmNCa=1gIOoDEM4eMlp}7sY51Pa0;p|)FhF3PP)q;e_j{B0?HD6@-2S|Ql+;6&>A!QTUmp-aakPh(JiTw6TQoD46$O5;M@U>~m#L?< zzfL$=YoT`hw;>cMft@dCF|kyhp)u&_u%9YYy8F~Owwaa469Q^UDHYn(szu7=C=_Zq z2MZYj{O5e?lwFL-#$LgC)bNVZ{yKs5D1I4xGv9yahkSn_Oh_huKLWePiS}R$NV76m zQ5w7Nug(XLn9qm0(6}?J%^L8OzLPMd;hI-tB-3+VH)EM)0Kj)TM ziS1Yst6Q*nQxUem?2xVq{@3v#C3UrY0d6q9$pN9zLK^$)CT}m7&7E72bMANgJwfDo zqJ8v6a_*d=-krlgKik_y=39GV)}4`t6vk${3Uepa@o@#-qukXWRcPa_#s;A51AUiu zFY&xNWB2E6MDa-DCu|?|1;%|z1Ln^1Vj20>r`!_O{a5kLt10X&aBGRqyKMa3s?;Ij zz44cBc(8>xEr{PCGC{|g0;zrQJQe1_ueddkArGaayF-A*j5zl^ep2v+5yRyowL6kv|mQy4Y$~u8%6&sJ_Gxl$y+wht50Re zzQVZlnD?T>czr%vaMmYwf6?SSIN@AAY$K}|ET{fv?ov$cOd;iGbEkyQoMb%@j(Ro3Bg!InPy%NS+r-uffm;mS za0j-Pl#e{(q4{rDy72>K~Zs>#O&hTrEa`)CgLG~*qB$wF1Jg^98k)*5=w(Ad{JAaP5rpm2vJ zpP1peV*l2-V*2iM)mz6O@V~sTgo%l!v2G`R-Er&|KqDaka4e8WD^|!D+%J7K`Uq)) z5ZCx|rYL|~TF3)nL6x6U9sH07sJa1sg=AoryN~*{u>$An4;X0McWW8S&YbzmVK(&x zY>16kZ6*jRkD75|QHvcvK@lcP&C8_>yRi0O{nwQaWl!=3S8FW=4~G+1<;k{*hGVAY zD&n8+=O zcEe@!mec|5nKeBt=JpQHSY2}dxy)jhyG#uCZh1Ix;b6=&tbIl)D(dn$CsPJ*kfts6n4~#*EuqML*FWAh9N&mVNTT|ARQ@U zAOJYT!xV{CG+*fNx)=uDRgO`67&X|`&OsAFlmI61;TN{PO0K+smk=X{5H}AT{_)>q zQB8YWOUCan!M>1im7bn;fzo!rb=$TTpqTCxo_iP33#~NP_qbxCEjotuiEeu+6bc$( zu$?-BIB;1Q=<+U&M2mt2S+)Y16j>=?FqnV=KP4U+BruIa(gj&40V!Zyr7gu4b5zL6 zk)%PwOSxQcS_#PVZf(8t<00leL~zt0#{nh$qVZr4H{%{%OsL@ThJ?sTAY z-{=Qdu$;q|G1;t8sCmR4bmFfmm{y9sv<)pW?>ixN6pqX{kZpU#%%KYyD}q)eoOp&_ zwMaI`1*+{}_#j7s6jWJ6x*cxizaXU>X-3M7$Y$K(r?p~&?5^7qtYfIAUH!@ z9&cD^4R?(Ilf^;j%~GfOU1~h~Q3L(oH_9f`?3|W~c)zy-5$z#}Yuie@92rGK?R1^q zR2aV5(35lo z1X%hd;PQl|4Ovq9BFa$Uc=gL9VB-{GhMLkWJ#wj$9G_u}As}Y&K6$dkkUT~F{gyF} zeT5T(q@F7PRG&MCr*+Vp92!EDWXY}aU{pW`#0qcAuBUBk%HjE|554ygD8?6}ScC0g z8wUYN2FfY*jx^zGIz<^otU8~pI1jzU!*8>w$;;b3jIbU>t^sG{Ln3wcr$LLyU=XuN zMPuK84Oc*17_ZtHeZ@~wuc){}`vqWHt=tpIRwS^nfScO#w)@9_@t<=>vT9-v1y$%h z0xuld{!7EZ+P+Kza@dHhvb6F_7`7h2)Cb+R7R2$K@vqqeLS*Z zP%%4Jz2}7oAtRe==^n5pK{k-rEsn%7uN^_cD-G<+)>IfLHmSNQ<#F!QMe}E4)YKzt z9=|-W9sfjn?qx@Aj|pe9t|Uz=bZu&u^mpdxaUIm2G6;McO4zgIO5T|-A8;VcKHX*> z^|>w<^hK|J*tF|Kf_!F!XcI;R7`n%6;pBLG)@R+X;){KxqbI<$WEu}AgLkTMCDDtF zz^{a!;`s7>L|~#z3x)q!@^XlhaL{6YtI^%(xPJn39l3S`C`1=;-1}z!FwPrNwi?V? zCPCZ#QhdJ%*_->2(V1iES?2SH0#jMpGrDr`k!*FnOCOQoKo#qbp4KrtLmE8 zVBQl~G504umV6h4JHP3%Jh?ldqWc6VP#rwicOhfC{qw_9Gr)z-1M%M#T3#Rj0F+ZU z^I5eMu$JtT2xDpIMLR&YvETumx?c-;hDxKx6HJpl@{n@C0GmA@4B5`CZ)iHm)tUb5nWFFDh*`dHPQDmLAHed} z`hhYA*RXAzJDULUAL*uFh|)z;-LQn%ZDV$k$OrV6RhQgT43viw2*+K8yx))s5grZ) z*lm0^VhX06CIMbsQZ8O}{>E2C|F>SdJl@_jNkTX*l;C zLIHA&d$J_6M!XvEgRx}*vNTJ(7d)eehfBR#8lCuPLw&20?wUG&(a5Ac`QY${*n3Q_ zb55GvM#)pH>9+MJlOZhHp_;A0D*k}HkT}SH&ju$PB8s=C@tU=t$z^>bi!TjHz4wt- zk1_e&SQNNv%S5T1>M_gywlwCWYS5acmrN4;K<=2jv0a)b1*V0=2;AjJ~;qdn!B z^|r&Q9bTf+xXLpG#eeY`%oIg;Wk}1&P*PTR1(A)t{kLlQcZ_rZtJA~11jdLRL$9Zz zxA*gFpQVmnZQkw4(UxGrg@|V5&Z$%Gpi_)#u7srz0j^He#mf$liYZNNk|qiBB#ZxH zqhgNAz;-0CU?9^WEtOpK!BP-BZD{$!|L$~wzkrO2EKQ3_L8rDt%c)yGB3&N{7g~49`?z6rlLRBn zrg%*bD2S$qht;H2ONH*Tt?;eZ#A;v4K$%XF&j6oL1H?yqVPSCfi$5uB(#5cBq^}Y& zQ~mi@jEL3t=>PI7F%U`mn}o9_Qm7^E(vuxzbnLPGyNxpN!i?sx&KJHdshU?!h)o>J zY!@zC&A$&7K}^FpN?GY{pG;eto&DIGAGG4W!3e%;pDV;e!}_m&b;uU}Uw(>0b?*a%Qq3H{n9Qf!RCTO71RSP z$%eRuc$+A~GidE|$bSowy}N}QNHrri@jU<+5m7v>ol{8o;;F~;K~n;DpS(4ULlfZP zOPRcOZjo*#@8+~ohHwGA6VTcGRVpsOV}_*PA%4i9K()wa#ovuHmL#oCbu++MEu%%1 zkZi{fjV)1(HgPt-5ux-$*+q4q&hl%voV63yqltKQk}tN%yhqrr+9l3c4VAT-M)_DS zW7++~g@S;3(hh-z)d8N!=ZE6e8$LQetxy^&fy;lRtNU0 zM!=Lh2dWnAPpC`jUhu&AZ!)Cn@R(6kN5NvX-HCN3P(jy{1IV7T=zVwaMU_1=%R7z9 zsFxYiZg_vALGbWpJ-Kx`^VC2LBGd(Txppe}+SDGU{ZxRk zxCUnXLR4I}JCV1wQhD}|{iPK*0|nJ<9nLL6RTNe-nG8decWs1uR=fFWwJMXzW*lCf znz7UQ670Y>;N~Fht5Y7JP;l)IH#N2N&m@gV>CukMwm}oHQK;$2jrWfOU0d|ShY#C^ z1*eb2!53E7N}Xgz@elt5AYI=NPX(lr<|e)QFi4OfEb_%EfdqNUn5M4<5ux%X$JSUtAoD=;o<9XKSqJuNepzc=*3u;Oh^p z9Q5SmmE0rw^laGJ$6VZf8A-|KhY}(p9zq4B(Ad}31nX!uN|*098-8`~sTTk2TvlIX zUcf|>$1%!+$J93^yqy~67rKlcRsDE!K>aYk7p0!#)r;cAY$~twhs1t}O??;~C{43d z{n*z>Di8}%)?+0)w72K&JhL=SN%XhlMZD7Ao*+UW#K4yG(rc#}7+c@a(3^QxkJ8>J zdJ9q1L{Gtm1JbquRInujcvoo84A4+nO&QZC5nx9p{)pIDR`IuPXi*)_F%l0o7?Sk; zi5)e$ruylro})g;;zQ;;qbwML1b>wEb%Ke%w=?G{C>0G~Q;?sG`BCX|90`yK^A|5< zdLT=!lBXvp%PWFU5*~XFk93|-l#JgmHVkgq=%f=}BHN->7oy(y;)~!h8sA?O9#a|3 zi7)2|2QzI#yt`(%*#3p6#hq@=!Ldi)nwqALx81$7N%AO0@mG4XwHmBjPOAL*l(viI zareH3Pbp7lxPGrBk=5(}V50e+_g}I=Ix86P4lfRUaTkyP`ZlyJxe!UY8IYzx=)+qY z(l;PO)yP+NJC&lYrN&yJE6(mVjeR!-mCFtZH8q1>WIzOsZhFIW?6Bn-!gu>OP#W-A zC20B17ZeIdG-5{$Y$;c}K&du;aC-ia$s88eeV;%nN15kYu6DfNP0q4keyrIleq!z% zmOvy`@_obpPVVpz-ff>dz&pn59!Ejc|0cLolr_1_v_v0X6<(ppR#nO}(O=Oms%gwp zV=rWa7FUFPQ!%3XUyICIl7py{bg50m)V!Wd_V1TSCy`c?wHlCbu29rDdZch;k`+`4 z*Df40`7dF$65WOWVe*a3o36q>C~DBHy3P3l`>O+^;}tX`BuO~_dCMmb(deVv-=C`Z zQm5a~W06Y&N^7s#b(<@G3hAfXgjA;BiDavxcpkK9gOMK1p;#G{8Yni;z=*EeLmxj9 zif6fMTtQ)9Q@G0j#X=qhukJ!r9vI@sDt>dQuNSOLo9ZISWqKq6K*~Uw!77n0lM@Fs zWg-}Wt|#@l(wxJ>Q`1wjdHTAVq{ti&o%1fo{%$?rWyLGwMqkLk&(w1%Y)zb(NTRJ{ zX)AizGB7W^-Pm>^%6FLneV`%vDad+&>dJXIagwV7mKCLshX}9v5Md&B=GX~jc3q** zIq%mLB?NOo=H#xwh$Nk`aIW%I2#d6=qZwCU(ZJJo=SDxW?lSQMAGT({p?M6pe``>p z9PxOaDnVCyQS5^0gCOmL?zAA9YV7cmn+B!B*T*kJ@#Rt&1T7+dFIN&4a9*CxjBKrx(PAUh|!k^KZUMHJ`r-Vf!Ip9UfUO z^Hk+#d!C&z00%Cp7DvR2vETBzox8SB^>cQzGf0-2D@2Y?cH8;iK~>l$DR`F2?2mPw zV`ZFuSC4&Fy*K<`6QI|Gt7T>}`wT;;WDjzANb*WWwnD2Yj74^eR0r|^+b#gv#NmgGUm?$o3V6j7^L@yale-Mry*m*8F4nFML1bDW#9;=7 z7qb19B1C63C05(!D83?rm`S%kmwM>C(jSmgAmMKzAa=%!gz=f}T{McyHpho!@x$9= zJ{WdX==P*Xe3gRS1U2Rt6{nZxpmzy^#~ZPJG%`H_v1Mac`d*>s+Ef$L6McmRLbRgK zTw$>nQRt%Pv{IiZ?$G^l^Rml$W1ZZg+49733au3Cr5`daz!EUkNiPVFi8cb4jI zBR(&iDEL1#j>v8cxNN-JqRdmB85sd6PEG*pn@@I}C z7Fply{*XyMRjkHf^f(2MH|=N#|M8?x?Qg=c?E)?N6VFf3@~gq>E`2-}cz7NAV1bbY zrN28qDs)_V!8Ze<>0S;ZnM+3Nw_*0-1ODBXgkrQqIwmai_UtxfON*%|h*ut1&n6{^ zwkKImCdj8M+=Y8(Eo20UZ9xkgtTyRe1MWoBlLVhXT=>d#1+MW!E+6DnP_z+!?~k{swu6I#__WsTrUkca65^FPmUgy&~!A>HsQ>f(b|>EnB;J- zN%j?jq2Cvnnx|;+^{f>FsWS=5t3Kd*5JwbjNc}>#xFV%_R=zc`%I2xhKVM1I)91wA zqR-F8)T!MnesSo1j!jsKfl9x-ka_?0*Q&oAlLnrc(M^US8JWkk-HaJu1D9k)M6Z0u z+s*Yb6+0C1x9)9^uxq90|0~5vG=IlpIvqJO^ePgL>1R$-Cm$NUK~zTzW1qQxx+uvY+n%fa|~f_NBo z2#CyZvLVP$;u?rkLiNNrjluf3{t-5A@prf+zy_HXbMbpR1o5ywOy(ddvr4WckNp{_ z9JPw}%yqQlVPX-I&Pfar2%88Um6-U0OJzD5)>xVo%(mKe*>uxLFQz})D4wy)yCvyL zbzTSW?d6Uc_VtS7=5ohKehmoAlvz_;oRw`p^Ynn^vKV`7z3eXWdU4?caaBuLvb%n{ zp|Oz_b;E+ANFxo2K3JyWON52O!MZy3tm2m=7$ZK`{D@xavgt^Es+@0-#JUNlH|_-C zz;>-K4SyYn9=uT}HQ;HR{ApgPr1*c^@&sME%xz{&alnqp@TS`japzryoP&i&1-M(L zJ;8-N_$VYG!2yyHz|FTCve-LUC*tK$IESIzym~q9LV~yT61lm`LD`~-{s6i{QKDC{ zH_Q7ThG}Nzga#Pi%WALQG$s}-V4VKSUts7Jmd?BA!P@+oLYw#LgShlg0ZZZh9hD*N zInK$*a|g=PVor7u4w)>t;EL+#a+SZ)SqqDlCLp9RaAmx<0S206fFu)30N5a?%I|dY z_^fHC?%eK-sK`-|E-4rasC;B-?KrXBIU7R=RSUSR77g-Gq@A6OJZEKZZl5-lw8yZu z^{VI-gizpGsZSg_U|;_t%pB44T6^LFc*0bZ0@p?&FD# zU}23VBihayOS0P6mU;Mng1tH3=datTN|*=Rscc@=Vh@rnqn|S|HPl1N$1MnQF`0YD zbj8O;oZlXsk~_BYod#}`{RqdEUlGhSHUQfW+n`(og7lB)ud`&HaIJoNY{3dNz| z2}m9n6?uCh>JU?~Yh@yp6suho&u@@_buSiL9olOAMS2hHD-fzWWz<@HYmaju90^P) z9$1?8N=ED0X$&Lx5WOde{QyR1#r;tk1mH`7QLVR0xrY+z+VBm=gQvXD-yik9%$+x# zkLZg0r*>^nltivywo>eWrR{jn9i5Jk%ky=u(>IIFXL+D8~@i=I)uq)tVJ?Cvy(X9Tlsg`}W;sd%j<~9EiPuvS$42edtL2 z>6=d(r{>Ox^5k2Tz)4^V&W<`@oAp1cp1XMCIm3JLFt#QxFBIfYz>;fb4%n`%39?Oe z2hXq1?;6j@631hof&|s|kaKC*Z9$3|ruiTu-))q>DRdewh0^g(;?{ckvyG7dY%tdQp4L`x$Uw)xj zPC+fNH0{cQiQ#@}ev(5~;pTDiql|PX-Gi(bFGHJKHFThUwX0kBcel!JlplBiwRHSc@*7EXqiqFSwoc@YwjSp&^=YNbf^P7#i&n@TNuJrLQj)TLBH{ciV!;-*3uoY{jC*a; z#OHB=N1qAE247qyen6jN{~Y7!m2m6gc=w$r*vv-v>h7gqPN`pYui9LHO9=vS5+1N#4w!0>S}TEk_Ee}t!|G% zUDdewHXXy#6Jl68mOm$x;ccGS5v{be5Zota+cQVsmJ4zEVZyXZX_K&7^j=+Lzr!AY zIAdQ%7z?Rt`99)BcEzObz^1#9SkVP}mQT_AuAxR(H=m4k64s}A5iRMQrR@1(LHLBXP=W$Oa30wLK7(mX1^St#eKW=tv7=wsRlK%gg@{H(;8(Tl1^?i`M*D zmNd=igIQ@xxhHuywT3gggkt1|r2J@LIab}LambVsTr3>F44E^Ky?*jzM%P1}3y(Ox)cL$4EKtA>je8@O z?s!cDgeRK+Bn181+u!|KesCeJeRthz%5unajMidR;dbs#O z=mY~cMa@w5RFbQDaRg`$1&1I2IEt9;kz@h_bcBZ$I4jqP(g|lHUVx6wS2G$VoIMxR zLn=;P_q*louckp|&GY@MYa}xUA)Be#fPQ23*H^!+L7m*O#|cNIUh`SVdHN|xA&yC8 zzsTIm|14%bhlylGewJ`m0fhnHuuQcmL2kaC=OI4YsC1dGh%HbnOhq9!biB7FrsdU^%F&P{H(WI_Jb#UqH>jNtlnfA%j;i?g<`+q~ z{6W&@-^^b3I`Jn5$r&WkVyq5Z4}!~ z^6i6m33+}dAbcv#%wp77vnoRUJOpaDTR(pMz`?egDMZ%)kgKLYVJ+J3#c+ViyCZr> z#n|f@-p<+`^?k@10~qH3UJAzAm7OauP#P8%Ihn{xTRSlw;;VO0(FX9c-_w#nXDht= z^u)7|@K4p-I8Y39ZUa!vXQHfdBxKpqVXR(yAz(SKdrFeTI!y{y?txHTtnCKclqvEcYk4 zU1zAgKMWX;AZqBx?HH^Z7J(yW&%m)~sw;>BtxFv4~J^HLLt-UJ|Xd4^JNn|ND@Bp~Cv?x8? z(y%V7FD^FatM)kO%iQP|?NrO`1gXRngS~D;B$B#)Cgohb9#gYG{RF#=G}&e}qlSO4 zFS#@3ikJ#Ot*D=W!c$`}%8e1+QM#EKxj_I>($v_rWO6Nc%l)8c!{5fPO1sAad-J;{ z6|y;g3h$rle0bWzPR)wzA^_5VIUNW+Ap)EYCX3Ttyt9r1OhG*^34M#1Gv>u8n1Am% z`Rx)^1L~*~HmbsOtT|p+OJ{ZFxxIm#=yTfsq+($*C@=9%DW~?~@P{>&{5G?mDa*{p z?N3vI&H4XjPcudOAhgEG*oA+r=am(k zJ!nt40$P$s_l8Y>dQIT^?#clOwLBn*M+!x^!z~gvFje+7rS$e+4U8JhQvDjua~0&^ z)sB0z*eShLwb8RM3quCa1m{uGDlWTH(CUwJ1H=JHX_cLq7hWJx5gQig74*l!F9uTZ zQ#JVOio=VN(y4WvUwJrQ*FSA))FqC<(h+-04h(j{@ZN3^ocvel9RaZ_5-Be(Kpe!b zb!eHNjYNM8O#F`N+oaLi*N^dVlVhavp8R#;AR3!KAV3sc3JOw|?n~Z?Pbg4>GQ4L; z=k5ungq!-iEY1`Y{q!L0@ zgYI45>Z-A>Unh%xSFmCfztIl%Eg3k5zP^`?l#$2n z!yBgzMEettX@Et^UAlD1xIJ}75mqiAGiwd@|ad(xF^K!od#t~{EIDptFf2S3v<4D?}Y6?$jaw+Km~`JdV){%Hgk zd}9kJB){ZeBC$G%2c%RCXeMV6=ZzEtcS|ojPcgJML+t9``}Hqf&a_=1*Z|n_$xCCb zXOOD^aap0fyg-pDAOnmdnafz#ab|b8p%Gp0IG?cBVGZ|zaCsgyeCPIWF9II}vt;L3 zSPjb-NEF*a4aEQ`#mM8M!$h$mP-QCO;3gIE7vTR!T7z}j!`dCPa!5+C%8k3)v1_ET z97&++4in%VAKcIHFUYvHAEY(Q-32yXaA3Y}Rdt?Ri>R~VR|RZWZro$QSBd&=AK}2){!f=h zcwZ-@-i^SekxBNZv)Mmg>FH{Jo>RA%HcYaX*#0BBDL$%u>la6E-e@rO(VlU93+Oy! z06Fs6y80h3$>x#wKv}VF$&n2RluqD%B2S9}E?x8AOL}7g@9Yon1;!ZWk%GXUk+O2N ze6s5UZnhi!OVDmO>@a2DDh=i*M=`$WpVBZxnF38^Q1EU%&Vs`_MA|^ajZ0I))Gg9X zlo5WQDmp|jG-@RRv=SkKL}wVpTDYW(tAHqFSpmBZT7SHV3Ld0YJB_A98dcjA1Gbz2 zQ29WBo9WtWJN7~367nAF!zY*K1ZB%oNJ-c3Rs7Q+<Zf!t46fnOj9*FFvY(LIkKwy%9M}o4dDm@raVXSXwvZO zYL6~!i%ASq^FFxd+h~cnk1Q$)mksn3>VW)GRwvID`3oR{rexrK{HH;Z7lzt*;) z_#`VlVNrbFKKTa7ARxkUq*XodBv}mONc%(FH`?a_GR}Xp4u}#mJzNv#zBhnwi^1e6 z)&&dP*rpTQXdcc)gxDxbj|QWNdQvt$$yt&CI{gH`Zk>bl%VFO~X{OMlV)Bw%5J2Il zly)SISQD#L1(uH{Bn(<=tuHQtqajweN<&hqu2mo$@Bq^tD=({@!cEEsr0z2B=IgAe zc=BHWJp%n=Ku5&t0%GZnZ=hWccYTn}d64oIl;Gq{sn-D>8NXi-OwATfKpxwUCpN^AX8PQQD(O$Yn6AwGe%xu%1z489^IJy?m=&3yhQ0})bo1D+W?q5 z;UnqYFXP=U(u1jfc-Le}JvCQF+EUtp@}P8Hj%8}Mnbe|v7&?&4d$lMy0tRyS;LMLd zG!j8j;w3cjS+W``p!`kiS8_x_~~^EoUm6s0z>i zs8`K0nuU}K$g4j1nU7RL1S;0k@y6SlG?Lm(>8EoRw9awbDBav;;sxeHkplv^LsXVu zfFdxYyxAkxQZSU@pZbtrPT{zQzwK?p+TN+3tB(RN|M3e~`po$V$kngZUi4}rmTUu* zDidfiNg}%}U+s)*n0l{O*3qS%3Cu#=S%Y)IQT>iyWA8(0DG#pJ0ucW&D%TvR%lYlX zSDTuYkicRyQOen#eTwks1?7N&adBj<>hF4;*x4&=Ce=a3C8wO90T|$gw3*6}A0KaK zfT@ePB2Ba2tJd|Mf(iXK{+F?A4OX*)UkCf-w zm>ADRaAj}a#tD<+!=B_7EdN|te8|J8F?#FhJs8rr{v1V__sff~@Hqf}KyON&d-1KBxr zZC@G|b~5z2lib~{dVB`h_USI|&W54bpX|ARZNfPo7Xm9~oQ>b$twYdCM|${tG|G4k zIQ>eej;(kA!@pY^aiJcRn^ThSH8F`_;uKqAsA25~^%n3cbs7u4c*t6ca&`k!bvel`Vj%5R`R#3xc@A<6+rNU3u z)T7d!{bhQObLKnk(Fr>PYRj6{;e9(|fe%}WS9DgKEf@HtyIH~=z4R=UsU^j%Xd>iV ziGkq|7h)bd{bm^2aLYqi??B5L`6Ab$`kd%=4l2X}47|2~Ci94>iD3X6 z^NewFT3o$+nY{J(@3( z(Iy8%$4*fDhIlIdfcsLBBEsjye1q@a^(D|P`H(eJP}~wTOr|T2cGkUrJj(0yA&>Ub z0ok>`2-3eLnUR0}=Csh%gFacwfes(3B616I*Dl#<%pfXzcs;9DoHRy{>R@WlGtNR$ z0{nfb!aUC}o5iz>OItL8{6jluEK`+<&M&{*#JdlLEaQJVMmj!Ed5{PirNVJZ-N84Y zubcU~z;7nl^Gt6m4hd&31J<&zfLUp#WHEr1UOt-V@J<7GnAgZ0U=10Mb}QK`l~{**H_EcxU)xYxh8c5vRN{qd7# z)mY_bFD<>PK&boBBM!% zBW#q5N*xhl_ZX$w>wCpN+NB{`<9=ne!X>qC%9B5=KLFqILRwTWb=uK0uCb9v=aBtR zD#8YnM@I;s1yT!%HPT5}9l7$VjJ}k-+Rr-8GgDg5-%Cuk=5b>cXs@${ilv_^g*$NIpjUiCI{ZwB!|Pifm>Glt%r*@Wcr}+KF_2%dNwhE3HsvsOS9HebqKfd00|d>jE*KfmqNalPxNwTt}?; zMhaqwe8zPS8(PXz5)7@|o^$%XQ54W>@0E2OT$}0uJqHO?#k1v6Y(zkBj)N=390_M| zNca`jP<>P6jv`)(AFR^m)u(*MLDGQ@=BZ@vreBq3tJx|7kgu8%f3Na>Eou-8)# z3)|hZUBo+5;+eLyH&ZS)h~r$LHej8!75 z=?6K85HST@ExZ70mzR6PvYd9tbdp6X%2*&ls?qG^t!9JS9wk&(ey+-`U?GuRY?Bf^ zh1cTQzn%h93tUtybdPt9_47uKAET*R@$KiSfnYuC_%K@Hoj#BA+mKJ=RMNq*Ye3zf zod!94>SJ{O3)2G$d~%vkVTImDE2Y3@w+eI$VSpPVs-kw)#tFD|_J`dPSjwO|x?B5m zo@4z59X0Sh+(CdE3|77Vgyul;sLrauf_(MeIxs`$0R}e*%Nq4EAQPgh#mL(Y94|KY zUgoHIs}9DWG8?=q7*-EJ-7RxoH1563uPa67V0wFY{)OAIz=cB?Zsn#WFb>?yDhs}z z=|9Uod7c0<(AF9hwkftUBtwQKqku0L+K-(ueT# z4=$YbdS&F~vx%qmx&0AH=J;#ZJ=Ze;VAj9xwI`re2Y={!I60FA{!yJ`T?70~N{+cX zIn&U4V@Bb_>qd@%bbNJXUr`LTSso8)f3K(3$NYfBrvv#>TzrX?jT?Y@AE;<}H(pRWwigv>7sRJ}4$%cB zMFl5U#*9o}MCKb;$9tgG%53JQb88PhRrdOcUrD~G1Lc(W7tc4gSh>tMY<-C;=eE%L z*n5$jvnu2#9sPF)U+SB_z89bAmFqi&Z+4-3>kJVax|(fPrQ8YDWs8|6GPQ=h2YnuX zIgBQLwzOV2;TdDLat3m- zMM|AHvBGh*+oav;y)E{=dakaK>NjWpQm3mQ7~Co{L>o{yjXPZG{b!?yK0QJz?=iv| zO{XOpKgF&=umpq%ViRD|B}QO@?eedJ>PwKO68>upCCMRdHr47R%QMH~DL%p5ErPV_ ze|q-(%@vTsC?alKIlB}au*8?#rl^cvTX9?w-1qn{GV*_y{1od`8nZ*pOC{FVC(rDoYTxJp%u4U69w4` zapR&e{L??KPg;Z#e#r<$KS;Ae*@)u#0U~8W8MB7`RW-#lWc5w?EUHqc(a=bPcN(4+ z80rtiHyyrP0bu~F6emo_eSA`w$P%Tone=Lvh-<&T1%dqK)UeplBK~@i)G{$O zsW{6{KHK{$D0p({yiwwH(L}<>d%DtL(Y^y;EFx8dI++V%k zh=OJf9uXz#=xw?Y2(#*AzY{l_y-&LAto!FrWE)|PWtv-T{`~tB%z<{+@wu%pNlwBh zEx%@1)A=BfGK}}nAb1^Oacg`70WVOhP$mZWI6uYQ8^-$QBRP|#&>fp>2NqNHOzS`2OxDizvtQdj(^;%GTgYh%D0Qaoi0zvl$o|zlMtY+JhE)~6I$TbdDCnF) zpJ?%_=ELVu#=(Y-aiMFqv1?Tk@jI_AUQIlB_UgW&QCPMl7p?Q=8GB5S(v1Gp z(DQ-(4VTAu<8Ien_jd$1PBxJ=qsP?0kDW-yf$Wqu$G!pG9Y2`Cw-6?)o!6m=?wpw1 z<+MvN@o3G*)Ok0d<13m(QeG7aaspD=Z<~Gj0WiS`)vRbrttzu4vI}6f84~AegNOfD z-6MT#>#n{5!7x7<15c2sZ1CNG%SjfcxTdjx<*DA)AH*9{^zpp+iIaTeJXOZoH&-D; zOl?}-7a|Pnv|uA&ImVGi#_pmvbkQndx1g~-&+O}r2AMcoJGW*y$GmfKFExfILV~js zT>QOGXXysBdwSfgTn8uW0k9YlY&^Bw{d#X}r&af7YUL=~7W0KgVaOR&7A^qe`AM4q z*oY|qgbdpYoh^SRxjWkZ<*%OUgSWveihCQcxi~pR2%a=B)wQYod!qTpIxN#zb`Mau zc$RlRL9c^hYFPZRe!LcZ$TF#FJ|F$;&ZyjVC0XpyL(gN(cDLHh&_rp4FU=x2`3e4B zKsq;};5;|MEQ&eLS9V*E>wHqzms{c&L zMG&JxzcLByH}1mNkN4Dqq=uSqTDRz7v3105F6}^4ZerYgQQOedN-rgu8OR0xB`ApW z_uHK^kRIum8UdYr`q_TO+&@tJ8k}3dKWBs)CrwP1n+^gwPsU9qjE?b(TDd;gy=ju4P@u6{>d5N3#~Me?GYVg zZgI!2y7l+t>aOgTHG<=zKeQ!eM`+mg)1}wW+a2utabo7YXQ>3_b$6! z0MZ9w^5$Plksezgv!g@n^Ma=qx>xd^!3W@PzVhNy*APWC=$SlcAbA9e4`$qKHZ^zz z%9j!CQ1)c#s>&ZZ`{E&r`Lwv2Ol5$27=7dOV(<_!(KDth+PeyFPoq+>cG3@a^p&)p2mvluDO3^dJImcb&z+7GAVmCLmVI0Fq_ndos>}PU ze+I>V@-A-pHpFjKtY6|kz5J9?G{et&|9dHzC!8t}Nwu@zg-J;wH}kjNTSFCft;en< zEnl2sfalsqCWU>REg9OjJ~C-S4dAL{PSXB9CpDS?E%xSX5ItF5SttA3 zjhq0+{I63Y&;KPc+}o7L{+3VTmoZ!h{{6aJ7=6 zIO)TR!T5KV8_hJ`09Ii#JW!kap1K2M@#pL!opPniw2+wgi3ee=;}8l*TP$H;%>;pi zU=*8n)geCzr{oab^|Y$+!}niZSgD z8f8v#cV5g#R=H>U%83kXRWWJX2?;&VmF9ewztWRsEU=aMdBXw(U`huB>!5mKxcE2t zF0Pu+mdHNMVq02TN>4(HC_w%u#t?spNObUAHr{FCbc!b6%4$%oLLMa2wTsOe`6w$Q zc+SlKBh)?ri6?AFGF0hfoAsrMJDz$J)N>|Qc9m%}&#%^YVmjXd6O= zC+^L&&#I6Ks;1GF)2)NSpdVBT6+D;{+CuJqt9zhKfZ(bu_n#dEI|LWru1DaWY7=ue zJ)El<&Nt-zXS@By&b%pNQP&DCQ)xkLll0X}TcRusHN|8^C*T#|EaJ zM!eT&0%F(r@;`7N5zK2>xf){65wVEsO}n(ju36eJXUwXIRchTHl#vaty#f4B2<(f3 zc0Fj->mmL`Q%X`E=W6l6+LVN0hin&7PjpW?LEN96jl5R-PR1N_<@YswGBZxs&0;2i zWd*0fiuyY^%DO(exvei3slXd?L-Qjq z{DOtvs-HuVVblimqN!$Hgx)JqhZbEN2F^erxiz!~V$X>a29T4SYXl+DXX9T_C;FIc zU~wyp3)_ME;=f-+wj-X!^Nx$Jmljg8kVJ^AZIUJ$Q(& zTp8HdxPJ?vTTc7)aVppP@y#BB^j+<>=;zlY|AP$pxL{(o#5}|3o00R#;Dnqrw_gBo zpaoy}2r^a=!9}aYbN1|St$>TFT=U)du;K3OF)~hdvaH{xZbB)N5c?t4R_xl8lA7V7 zb)45jzPePT`!Wt}#CMEo+d8S-tx%}Q!|Py#YrXra+LytqRfV5jO)6iUe!N385k6~t zgA&?_49`ryH6{c+6n(VFWZ zAWjsX9W84x2dnylV;bNd?7w$tDoO2&l)Hcj{l{U-J zv|N>je+3O={dG+@DJarFzeOT$%3Rf2)k^7M+&xXvXNMD`OF*NGyh|M3YDNKNHOCbA z{cBjT#gv`CtEkWwXh{Ew-q{D|aA%8;i^vh=G(GiMV|?xDX4~;-w&bDaJsBJHq&&81 zQ|-BgpoQO?T%8GydFanHWO*<9alu2Sn)o`e6YVN)T(TD~g${11{xAzJ?7n%n0K9rTCaWqHoh{bRnf?1U(wSD9}JG!bMn-iH5g=fQ%h#PGHv_iRZH z(nB}^#+0y8wPewX@yg&FR+8Ogm&Qg<>R1{;IgTD4e*z`gQ@>?(7&)|(jK&e@S9;vv zOzv{x$OQ58$A%6L4$6o134h@*84Xd=zo7rJy_S$$Vh<%F+ntV%Bi#`kul@q5)t^%0 z@ADNLV*t!eE;4b~OwaIFnViQ&h^W+2IP?Z=Z3*?mn}?YT6Uyo(5h7a8C(Mrqa|)@ImqpD|zk@roaEZuMC0 zVMi#(LNdoiNZIVHo5!X5^AZH7GS1I=Xe>+Yk~gyEn)!(?ajr%8%tRS0O% zxWZZM4O|&ZaXk|<-m@3&nD*Q%-+R^XdZ!m2zbK0?6k1qt%l}l|!y6fXm{U92snx&Q znSdb~ImpfR?3)v7ZU4a%mg5qurUo1I4Ej3B)d7B*Ht@jy0NS;*YjQLlmz|5sov0X~ z%XaRIJCA+>-YnNoZ?1O4gy~8Etg;8$Mqh^ zX&7jdmAV$cA6~UZud&>lWSf{o+qwKq`{A-@s0neWNMWrCqK)UuO2%vxN$`M*bilL( zIF~YPMDyz-YB~bCkJEN5R7(s>-{ue88vULYHC-+o)IDRp@dnf_G6Aqw0>n{pg%c1q zgZ%zVLU{(*JX4F(DmG)}BvA1Vmm#gHn=Q^Pe4VKr9HQ+ZJPwd5f3N@mEw-L{nf;>= z&h_A+L!(Y~G08E^CGrSRLQ;pc;gdV7pzUdnbX^813RIW5Cq2Ca_wVa}T*BYPQzCuLQ4+J4AG}oHu>~VC8yu3UGtqBls$~HDOYLO7tGd8Z&88%;9?QHAs zS6YR76-2&$AW(^_V+U=fnbv%HwRX3>6uj0}Co5Enp1z;yU3uHP2)rhAcMf0Zx+S}Q z9~1ExP4g0}=RclI7H}O|zez=vxxck&wYRyobPxE9YI}PLCLbSrX@SP*T`l(zaZ^AX zPMm-l;(0TpqoXL~j}4D6d+8-FHVthiZGcF}>ltHI8W6o$Oh#0ouAF$X#BgOEWDZ)a zK72|uA{{syf(We0yK=%-IElqRke8Dy9ZDGu!qHi`ruk|@yGko4nM^0NLrJHv%sUw2 z56b|rT%!gjRzrL(&96hS3eNi26(L32}o%9ZTX#_v5`Lcg!G_)d( zim7VXbk2{4`}F@<$*e}TXYx%azlpkSbq=%@Qmvvi4^yqq0m56_c1|yk_~3z3zQo4L zu3_kx%})vMENDzazc{ZqQ>HE3i{*Jr7Y<#Nh@t$v@~M!aRhpdFx8jD-g(1{v4=IIL zO3|i7Epu#Mz>cuzI@(CM4q=u4_=+O3ZjM|rbW6;OXz2BHKC$}5)`Qv6nDwi?>6{9{ zt7TOu7z2X!&_&l)VjUs(*#~D9gM>|96dc&Ne=8sF&R1b5PX6xX2TGt~L(|(z{P5UI zF^aqx=O6MxaV{661OyX^#JZ|HP)naO0?jeY5==cG?pDF;(ZXYnVm+%)jMs@8QQ{PV+tVW@_RyW{c@h=T=d=h^Hy;cq+OvzMQxS`}0V0AVq7^{mqIr_@^y`u+LmkM?E? zhhj=$Ki5J^clYhgE7o<72_9|B->CA^WBp*g;eMg^J=xQd>ebZTU2>*lj=bEIKD~Ss zwPEVHrW5dmbYgPF4LciXbpo6qGWcZAk}WlFHQDqEN_d}iCU7qLbTJ+*w9ZV9Y-znT zc{R-a#soy1X&ZdCvepNi-kNGVe!G@zdH?y{(_rW*)0vX3CY+lk6JOU@5AD$pAxnv+ z{g5c94v*{cS@TD`9=t|78$Zw0&LK1uuVEWu<%5@DBZP~cyX}3kW|lXYYqp0fw<9$q z8qNDcbvC=ZOU)$b4E?yF#hG+Q3nk(5gFA6OI!T|PY4PZ9KN*Bb z%KY3t)V7#(EDN!iJ_tfy3mI)$UCTdzFgRa!*?Fu9-jFCMstX&g+}Ef!M+=sPd=3+` zg0i8Ro@jXGx%59hQB0aX@{h?p<9cimOxPX$b)_ARXnF4na%!cgRT6Y4s4nfD zs8s7)?%X-rri?nOwEVgK>{*}Oe$j*enWHU-gfZQLNT>UC2?g)*}A3?Ow#5**~S|Yzf%98khh9)kSo#}iu8GAtfCBWhhj?b2E0Hk}$k zG%Jv*D7}X;t57?%L8n}! zcV6u&pY>|;y%iVuSS@wzj$X!xSQGbEM<$3W60Nm*A>P<5W+Q$r#6sGfN}?cbGz%;j?u>uV}$0qPJHy zsuZ1FVJ@>fydDyB(?(YKMFNEPe@U!T*}^1T^27`fjoc;HbMBOYM%lzu6oTw>KyYv~ zqZi`x1B;uT&&s!qtUu~Mk-PKE-a1{G!q%*CkckN4KO%vv#jJk3v%!9sy9#6sGzxta zQ303&p*I%wGaii}W0UxKI3CMh=tZg5yV4o-M!?JaD+JK%UH76fA&Ha{Y9i%JBZLTU zXO@T$b{Oh%Y8EBp7R|c1X?22~!@`rP)j8VjPF6 z{G#ic4PmRoSV^rEXcbX#|5_luEdFCq?a8B5l@c8m74GCB%ttMyyem@N={&8?{u-4x z<-82&Dk5y(8aFaEmPwMKk^N|kbR^XcG?;O4pR?jVyu*mpt?P_1e;@bE8gBBEDhq+E zAh9kO@wB~a=bZ>FhrK~AHZ87{K7E0KffQ9(G|w$x z5rOXyXozE;kt z(FFVK;MMy$h%=&Un41k<|0AZbFTY}126Vb0K~q~l@GcSqZQXv5gAMwx=UBZtt&nJz*})*{mdu)j9vlcH zE=e>%>M9S`wac)r`y^AG(ttr^N*qtUQ1$g)1@al)q2ogJ*Nl77=K5pqwjQCE9Gd`x zIS0sE8WbonW)`wohDOsFFNE@JU?iF+Yz(kGoclScfmML*Xkb!d{{j6xAC3?AktK!9eQ>#LxicLE&&{dQhkHxp7)vckQ#Gqloi zbGamx(zWsjhK8bd>#Z@=k~q-)-w>wLy-(RgOll#cy!vY2a9!u@C9PAWNw^wRzNcC@ zVOP?ox+Hmd1+A*RtKzJ)DhU>2=g8v%davg{w};PjoCmX4)GtUexsyl!@XJ~HViahh z|B-KjX)hX6ce9n5Fc|P-f>PLMo3F2ISKl{KwVO5nTysnzi6idpn8;Q>wBlpx_$FX+ z%p;3Z6}cL7Hw3Nmm^u-CNzlhwuEJTCKX>xZj3hrj96iNVBBrFt2e*ngJ>t3$c8rQD58<}ohFBd?g;>01qg9> zR1~`bB*pt|YM5=04gKnVflIJXt5}{Ntg&Xh^>ru!3p0oT92K=Oo(FC#UF+h1L=b*3 z7WG^vxWufC`tRBiVrdBI4L+T1B0j%p`5U5x8ovx5YHi|qPwbYQ;k zMX~CF6y1)S`F`c6B3A}GHc9=P1K+@{ss=41#oh5DJ+ zqXW9c8JX`rx)Zbwq&(EVd6tTfWW@%+3?_QyT~9Xgm)HR3`XGqsZ=Do`M@o6MF^{D= zr(x#7pNoV*-zHv{evwZC#Rr`Opc58AB3!&z17Lt(aNVtxO6z;Xyv|UVpgro{$?fL8 ze9U|)(l;or?C!`ooAq|2X`!w8>UUI{K(ak}D&#bp{v=SObkR6Dle||lrN0MWyf0;V z?{pRmIm5V90n^Eks%FC;LPSlHUZ?5{8oe?n7^`K|g---HX$SBhG(TbD2E>>=Y3hal ziKCD&@USHiBa0N9vd;dk^Sk_80u(+2BtRx*mY(-Wp(Rlgi}`-WOD6&`ZoGxy&49J5k9GI8c_^660jQ zyF~e>09_e*!mDbIu|)+XOV^Sec+6I%yMN6CtRWTXlhZ%VZ2$sx9nd3wh4*lC1$XP^ zKu0n3|Bv8-djrZ?L*m7t+ABatfrWr=4OCG81#%~??>%oEY>?*!6BG}VPpg_0b*2jf zjNPMxc-@`dIMN-VcjXb=@v&a;C!FBmD#9`~tHj_LJ|~1M>qd+pZ0{%kKFQAzTvL5=IMV z2Rk6|N%AAD7;1=|z>u~-c^mHDkgB<>^2!zl*!Q$JnT`+TzfyXUZ+M|K|HV`PW*++{ z%d{vHjD2HE%GZ`NXUbmjgl3w)dG9W(De#JF7Xe*1G{u&33Qb_rP6TfTYa#vFwBoZ~ zh2P9Qsk4%e@N(R|?!Mb^@#Aud%i6?IGCiD6(2P7kxiN_@pb{<8fC5p=tF*2h=uVEQ zsV@%NTnhmkOr$$b9d}>3!)!?2BL}rZ}g+UHSmM?f1wu8&DHQx}%pWDhLs)R-YHx zYHKUh1YFzGT8Rej&ARe9wA<7$2Z+^7fx=rmCURJ?ja-0VfwJLhWQaB;G{zERE2kZ6 zp4A**y5Q6+f57uljYtjq}nscc%wi3V_vOdbbqCYAhfkpXV$ zsa)jx8GRgQ=;vAkP@VU>ynlr#)g_Fq!6Lq@RcP?wrTuR%1X){neP_1hVxw8%|0VRA z)*bO)fnegQ)z5z$gyG8BL7v&cm$RgcW+IgKMms~Id3>ENxWk=_9==T23meFrseG$` zYv5VE9SG6~JL1~ro!k$91dC1w#;(ctQk1$q{Ql68{0TcLv}~lG+q=<-DSLOifDplW zLAW-*I)0YhT2K{aC(4!9Q>NXf8@G8})hebK;Q@Uq(|{vSpDlEKTep&bN-+J_f;pLS z2@2*P^1popBl9rwDyYv6$lCYlax3#Ye$Y7~#?{V`o zx)ZwInab5>dT4Y{uA{u}{FY7FR*8O>`;=`Z?A_aZ5{}hvbEa7p_LJh#?u-mQZb@w# zMgh&VnUg&i_l*Tk*n)}DnPa`%acvV{W6ZU9oPGN#FUT@yDB);w+|+t!4#MiwPGStejIgPS#r?kG zurSmjbO>Af*+{PIjZql9zHR%qg+1eQnP-$N=QlqXlcVvGr;5{Jna2`G{kXh#O3z{J z2J@t_K=V8Lr)!rQ&CuU8k~w+#UM7f_kSOMBRmqa3BT5HAT8wsXU~&fc%?O{Ufw{f; z+FFCFX3@l(KtC{kHac6$?XrK4;^^-oi4ftBZ6OJ=^J@z7e~9{2M$Pd7T#(|)boP*S zjennAaV0L`(|tLIT7?oNSh+q&S^Nv2eQ^RLYy7^f@^-@By-brs)8#$Ey%(_s_12P; zy*c@a`_!H1&^;tj)>xY;H3KKBG%t_Q*cKa(w;1nL26*tJMf;T{mk3Hm|1_9nmRdrY zGd}mE()V&>9O3J9enD1}4SiODb@g1O7ni{#q1&5CwI-moAqC=9JQ+~aluhniX9FGT zug`6r$gfd8KvNH#Oin)=FOzuSkEF2QNeK`@H%p#YBc(3aO^!!;MXTzS4#jSRdqtKu~_z*zM3%@z0Yzjm$;r{pPPx)hlQA0TH{C?f@I zG1R`8PTR0Hn~@}`Wz2()QUh#seulVKHlSThw2&b2u#9bYDYe8fx_L+o(Q?jmU(y?-?Gno>Ozi;%%K54* zIlfa1+B0=r23;((q=&t!Bv8N#f-0j z;RkuM6&>Hg>xWlo_%vam)7z>asLP-F)o8z@fL7SeD3H|=p2uE6KtT6Jz89@3AN?;G zv(NMY(HUfTe0DQ@0F=x(XnJB~@bb?qlKl$;v?NjXw5ETghU{mv2M&Sm;i(8)yPg@Z zP-mB+)bdln5hR9^V0fI0xyrp#RA3w%njuH73&`P;o`5Vr`JGgw4Q91U`owkG4K;g=>Sg1!* zn)=kbbe^@<0GI6lOYF=h=ph|WqQj|ve)Y2G4B==);c9?>0rga$c3DDw7e^2LWpCD9 z6thfm{^9mVN{p**#0A0lp(UWh18Ek|`O_Mc zQt@tPwbZFsAxCL0Gr%+`-2P@1jpay}%e6^~1mmb;B6s1fW(#z%Lxb74e1jRnn*c$$ z@H)b#cb@3Lk25xyc>jF^cjMw-{fXxRXxNmu8hM{HgPT6xFLX30q{JSV0Stqc`4W-g zp5Kioj&46-DUtvQ2l45^B@k$Q^gp?!g9i4{@G~+dyaRudFNj!Rl2lt{yC!;U?IU+1 zsuCmVoFT_i>J*2`TV>RvjQsJ_5|gsSGRwWDWSPBbCyPgay{Z|s)4bvtR9_dO<}Y7% z3BDu!JXf0p+FTrrdc+d7VPTG;wg%4U=btp71gf7Z7wrvPszj7=f7`0ls%IC6G2tI? zHayBO%QucFCOX3dKi(Wrj2U^1GB+Aoj@0qB5g$K$t6Nhp2|D+r!PHm! zG5ZxOQqU;DkvrWYAlJd*7-ikZhF)EwkAIp~+p(8N>(X9ElG5qk!L)w$e$SD~;6c)? zD-|iTK6a|qf)t0)l+;aWmsye}gPD;N>(&nf3YiDDx4YQ^4F$zlG#Y)h zUgNr+h1(6V8yw0vI)LQ1EU;Ge*e(9?-b=!uUJr=Z236;LZlwHEr}~^Ei8)7DCIc+~ z*|&iz{>=-?bD*We(E6SO^XW%+gO(7Jj*kKumz_?31U`TNx|qU&({TTtW5hY_YAbvw z5FLAZI7LLE2p~+WZ5zVdb;C`qczjs#m+YAMvaZqemmSZd<-eQVsS#G)Yu|vxjqyU> z+RoUgRlGM>(-LjbG0>?dOivr}Hf#UzP3wVnHM!qNVOgFIFP`hQZ=oR!s?6VpQYP&i zU~&#NydY1W&djSMa#{c^TwTODHh9wrINNMeGZKqCMj$;Yx8yxeA5zo0s6;O}XRl zwUbVlxw&aC0!Xe)nOxqi-dmB-4BT`V0!!93U`p)n*K?ha0@|JLFS!V5?vhBbXVR2Y z{!t#?y#d}l6YJ{(5XAb&E9cG?wkwy%Zj1;5tyK+j1Hu;&1mtfAL@e{fuh0#U*v-#f zVW%KvU=k?bU+pd`P)?Q{FLG*Uml0}`mhK%*?$^lxd=M9>RG&eqp1u$Ni_x=Q`GxjnQ z5NrjB0y}%=xRg&v(yVe5<*O+fekbfjJaVDeY+H_b5k? zS)v~xADPNq@K&gdE0fz)eN5pJlU(7hTW{H}G_Vz-zqK9XUS!FX=70bP2vLZF5 zDOV-2j|G*BuLwndeo@8J6&A3!(@XrNsInyHr*gRTYvGhprMUqpVz?g;QVX{+q)wKv z*_FD8oH4eIF|0+rOrtJe__;+X>x#N84xFZ1sQInRDK&Mi404IZbaQ@EO62o}Z|gIi zmYW`^51&|>8KoIEOkR4izH!s`E(`W*k}WLp9un`t_J2+v5_%<3YG% z2naaFsWtTLIc{tWS|;Ub*IROq$R1?y067l018)EQ4!8oImhNzz>xT67&9zGx|2^5d z!3S*UcaI#oxd6zHLSk<7mG|Kj7Z+7J$L=Nmo$N&kQ_H<5u~=VaozJn?Z#^aW3d~Id zGH0rEguG@_${xQkt4$}v^4FEWAaw#t-VJoRGlVu^Dx6~Rp$9<9la5hF!)tfV9KmEbmxff8Jiiql8N3@WoAtSIC-!6CvOA^ z_Zy}I@zYe{%xm_t%B6_HGokDF6x;_A)Nf(FGqi_E(_NoDT z``88g|1f9#b>0C%my#lcJWz!RnU*ig?5SUgevNIIv|89eV+ZcNDm7Xj%ej9hkr?B1b~#5pl03okuNyc=rZ;@C94p|x38;0`4E zQV~ULkdOF&Gspu9%b4NMf&MCUE{e-QK{)7IFUzi!Q3W?{G;hP8b?#ZIHe}B@{($;} z=7TC)KDu0N0=(}rE|P?q5~_RazlVI!SqAJ{Be0=YfWg53pKiNzKMrZ+GF5lSz$x)B z9~1;mvmA&S=_`ZS>r+&7ZH;+}; zxl3o)3ei8>^^>|FsL65p>yTSY1j+EXI*(e-vmVz?#H`JVXPOgb939(;&LDF#J%7mg z<}Z94sV+sFNje^gWvGI&{x-?EU}$CiBl(mDeeoD+9hVh@Xb)Wj7lH>3m5LiSb{tyz=8bZ z)W1rZ#2l!=(;kQzz2ciGzhCj-uWIk>HS4@C6%-BUxM2fCR3o|Wr$3}&5x^uy%kAw3 zC$V+e$4hOg^sIT{0d~5TQFWezKOE0Qz|uj-MZmD=aa-5wLI%7Fg%3@j4%I&1Gk^k?!;h8JE7I@5vn9%08e)hyZZ$64?HVVz(vFcyW>&9J3|Gp-NhveFIwg z(5vP3U!I*ymq7o*5G{|gBmdWH=ceyjKSxH^k8(@x6316K)q}g_PCiPbrZDY?PBjb| z<1gA8jIIE!>s*$!QU#vuQ`HopZX0#%iQ`K|O|foams;jBX0YX;o}auXg1bbbyn%Sq z`HPutMO*dKVD(`<81qCbsL{tw6pvaba<`l4;-_cai?0NW%j`8?FA?Pf(&NC|3(~9K zitP0o=_{R}62Z;J&c=KOZZ}D*+KVBh4i&Q_qLo`6>}9!1%Hq2NN8UKS?h=p_Wr4+S z)Pw4!`yw;`BuhP_>lt(>(^vH29N!H`ih&+62%LtRTf=4q78e?>{H=ZG{|R~>M+VS{ z0P#6kXC|TktDh4s(`FiyyC@~fIRR^-GUzHq^ zk)3SEZhq8rp7@NZ)4htFqA>4I1vP6T+QlaL*!8p1LQqg~u#ko9j%}*@VqJiNoAIA^ zlKiQN}`|%|PwR;XPJrfqpRd@`_

w0!8I{6gckAeRLE|akQD=rF8HkWKs(}N;ZO@hJ8OvD;( zD|_$>Vf)%(iG#n??kl&7LbIV#NdCgrM6U!0%o|SG?=}{+p!(rEG1OWN)orgCh^SOj zaK(LWuYlBglm(J0m)fh<0J9F8WXl$@mbU>L9)labpZg|*^`Z1!d7VRcTzYy6`$?w$ zw#HBft{;`_E!;=j2-n|BuLs8+(nsy&0{w+`WdWxiyDWOXBg!FQ|7Gq%erum6a1X@# z=AWGcktHM%r6k zJFJd|oJ#eRPvImP?g~_UT|@$pVfXiV`X55T2LRoGNc{w!Eu)E1Fp7YRlZE1;dVpK} ziQ<8EKzw`;p_*wl($joW(hjk(GkLeWR{wtJVt*HN5VH=m*ZYw%Y1~ zLVA;m(97UXH_}bK7p9fLeZ#3GYd6%tJ2UrKu?*exsMgW(-CC8;rxXvvgcjFWK6+SQ z374?DGSZN)PK_l-L)xk#PkfMpOcO0G}xhmNM2l z9MdTovIqWuVUcc-3aO|^s^{1<+7yjqUl#v44z7;G!BK=4lhXA#hNq@fxPbT?eLQtm zXPUd>qHW=G*lCaj5|o}B7|`Wwmjz24wyX62`jvs+I)P^mA^-+*yqIQe3?3GJ?7wTQR{oQ7Kpif3q*OgQ@;| zIA^v_xzly~y0*#Qpr1GSFIln(T)k&fe~oP%~y6ElF3Os@o=G7Kf56 zyfj}*UgU!9d)Z%xFMYc=S^|INTgRA-bZWD2EcsL`_PP5(7g~Tyr{t-`*&zanBgPy7 zPe%r}huOH5#_`iVurPZhe7Nns?g_;mYZNpzd@2Wd3q{?e|7ZuZe82sSAJ74S>BLZ6 z;+%!Xe;#NFBSpp^T*9XT0apc=#BKKN1PMKTb_TXPXMH3pVvljhspzOhw0uC1$O=dG zLV}gW;nuuU3#EB4XmMnwj)p3hxO@)@Cp=d26YvMDt?R%I9q)X0{tC?vxtrK;gH>{l zYH^7!tx16`fyBxOzf9n-#H0bpO3o9upS4JcsPXC1h}Upxn!ZVD-C#c6veDkzQ+JgF zlyB`J8Rd#lU{q{yGtZPM3b`eLWWtATZMvkWQI#m1XC^C8pl2kRvrQ#-v{FERe;80k zh^lhPPK&=Z68EbmYT+gIRz@Vafd;<$CGd|CjR6O^?bzL0_W3I@y`mE-_VH7M)V~@P z*~*?4BqmlxqwQ8B9g&mH`D^pS#prD5>cv?&Z=jRO#DT3wP4cafW)`sZN-P+bO4kV~ zeIu1{i#5##LAErELKN0Bg%NsRdvj-+Ld^Jl5)7J?wFW)#b0f;lIE9J@^ zryCPK#~41>HC;AR0;djEn1z{&fKLig@s*s~&NKu#foS#}e?y-^%aI3avxZ zfZAkzmrHkl)QU@Fy>`2FY;CESj)4`O8^x?^#1q4DvvOuT0k1T(cvvQr zcB#|;*v3e;Ex?5AwG`dUA7I$jxSaIKiuV_HNUd8YiHG~=4bREi)Yk{P#iMa~Q)nNP z{g?}SyA%0NQ_vR3X#{EVZkPfZ=%1dt>WOAidK-8wi*?Q+)eVGiB}^wKe~dr~YFEH` zF7A%W0Xqu8KUE|b&yhPaq+~Eee|v=gyN>%4C4%>RMfd$ZiGZ-pHY1mN(2%X3!zcvS zw{73OOPF@D4e!i$ z%1QFh6vBC~eB6TRd)sQ?>4LF}}(pD837BPy)tdfHPS z$BQS1>SWg z83H}+x%^&m;62QN8QCym1z^$Dka$|s;%k+bXo)WyVlySNi&raNE^4lH_10b@?%@_m zHSL}uwS;}&Jo%~9f}WHNz3C}nu+<^#)RenmiD)&BSQ$_-;EfGjVAP?IsF_-A2VS|0 z4E@B7=D;WRXD|A7Q4SJ_Ac|&5ih1!z+xm8L9$>YRr;-QLe&*sXZTCJ{oY_s)CA(lo zr3lG5-e6AvOPfK1?Y4H2k(Bs^+6SetC)K{-@G9N45b#n~QL)kk#p9k974YGthe;&h zE~5?;D$aY4tsL%$J`|tsqHun?cipM)K=9iWP9idmw4$m+vssM#?i;%Ix{v*v?_Ux#nGN?{qu*rUAd2`Bup=q0vW0P z{O_4rbFLYkOhx2|aCN?{;F%>k#x3>HHaTpmsUnnOw0zsK4$IpT(%z-2qZZ0Eq8G{) zhyae`s5P(v1PPRhd&g-~AxoeR%>av}uCUmPE=`f@1ud*MbDswX5C6ffKKJ$q=nJ(O zj2SVV#XOAJs`;v7(_BCQoGQaR0gt-$3=pexuxBF(A5C7v(~S?314MD#iwGESQ*j2A z+-Wr5-aeZpum5W2y`nR^QhV5Km&K@HxfyDlIa+j8K3$q%pOV)3=m5pV3MfYQi3KT#s zJXkwhH6l51Dup;i2fk=S3s_01-flGSlT&a7o6mu1*aaReAFYvsEAw_`%*oiKBZF#2 z?F3PwGCd&89yI`wdCsAA`iVNHnP!G!=B+r+Omg=6WI2vmxYB2a-NqEk_C9!C__ zc%f#qXSUy6q5WEac3ckw97Dv;CfGk(GF!>qagK`RB2?^EF>)q|%JS)W5lk@p&ZhTe z>AR7He^@?0)%#<)(cH0n{Y)YAR;Jo4Ql!`MuZ{bkpiLfr4M*zT{e5IlvM3tdAyZ{j z^=H_1B<`xGqM+y{f*v^}i%D)r7kWw6T-|(6_yD|ga^fJI?ARmxVe)u#-!cmcXkIu{ zr}vs8G=0pjJ{jkb=mQ|q+2OZ2%+UfRAYpzKY#hthml;mr$&tskx-q`PCpnUn=SrCx9GMc zMq;zutLS;QuX(QIhk_uVvG(diapfcqs{n$|;-Cy|NSwQqQk;p0ixJFgazPl7yB)H1 z`_%VL5?@5CHc+RBvBRitJ4!od(HQX4L%Q3d5V2*?M41s*4|z-M%UNfT=xVJx^ADXo znh66#&9yPVN=%ojKUKi5NvHR&+AQq3n>H}7jOh}trZnc7W!jiy>9gU&64fWR5|v$VuzsyVyz6tjVS)4>JxmeFAi8-x4ZezkQD z1{^C++uIHvMT!SIM!FTFVNP4MF8d$g9EtBLB-G;EKUws34CG;KT@+3BYCT;@mtqVs zSvH8<7Cj9N$vZ+Aok$X~8x%#_o7&h@-k~`DH<2m=kEp$bm2XDr@3U@RS@M+{c zXA3Cze*KSn4%|Sf+L3{OdqzTygA6Eh?Egy~_V&BsMaD$d_3D)e-Jj4`?7#txrk*+o zG+VH42AX>M#`P08vu}O-9U@e|j?5HzoR$pEb`8XZPlwcGGFPG8!3U+aa5Gh(b5jp!RAo|^|}k1`{UIq zN@XIB3tfFam7eqlq~A&aL$t%Jq|_6Dmc37sPC$&S9n7exPwNZnwHHS&<&!uk-)Mq) zxekU8tU5HA@?}MAGZi1GD2|_q;<5MH{dysRgtgwHV7$$#V5BJSiG?o2>h#E8oP+r(uz2Cl>Sd-OiJo zrmXBy)GJNF7ozz$Grbx)wL@=1R?74FzGy#dme)MiGq-jM4qWVZRHtW>p~l)NMAOXY zmSLJn2}~QS%W`fWdQCdve5Sr*V#;n>w2@ox?sXeS+bc@SS4*r!H0^Ba(E&Dm@nM6+ zgB~z#7{8HIY~Myam(ug>{#WH({LouPCXsurRI*`l7LtfB1U1HEwA$ajwRHqjw#D{! zFmwm+FBMug_1q!m+-e>@evN?b)1+ZvR<#0tN_n=Txg*P3NoQy-K=Wg&~+#ke+ zMlMkOJSG}qh_%T>4L7lLL+jrXWa5w3GAfWnYWT^iF1H_(=?hkIy>^2i@OPKG@+c=* zU4m6DADavKY`+R6)`Vz}?Qm0da12>qDlAbJ`8jINqBAOm)Uk4W+l>*?5wJ_oe$s`U z`>1SnDJ&A59k`ZmTw`JCP8rS4$OdzPKtFCH8RonPViOdq`YEE>%RDr1Ei9meo_KrW zxznUtArOs@Rqjcb^yo*$KM$7`X{)?Vu88_}V=UWdU)|^nPy4-&SXZ^$dXtAw1W{M@ zQA%Nq=pHMXY~B0X-a>sLpIMQ!aO9}a2)bB$b=+3*U|hUf$1~-trGNZhE&QOUml*1$ zL35qLHt5Zy`(pfq#xU(mQuL4Nm)WWAT_a?*3}eQcG)Lln*e9QO$1oOzEGF z{|{J1obF<|C!Dmx37V?jc9WPWu+;3)<^Mi3?#^(TQl8F6)pLNgo)vM}ib2G3r}NfH z7p|tpz>*gixyEtxY|AO{Nw>Ms&&1Fq7E8Pe(x6Rp5dr^0(O@5OuK(={1M z+e`o2@tki^x53f2BbxPTHKlw*XM8@j?+ft|hjh)gbX{!p+lD+$ z9#YLMB^%4@dv1W;O#NB4OR9}^ceh@`yDp%z5k~UYH@7Ko+bM({mC*CngBY4AI;!WKpHZ;DD& zo5iLIdzU)3KcSk^%Tlj};(gfWtIjkFly5%P9=a+{9Pe&h*rixs-Fv4X5!V^(v=gEZB+N&|)$%&M>-9 zW`0XFzx?So_GoLoM?CzaRZ!}vBv*O-UYMH0oO)Yy+4{lAPP=1A`*yLMde(~$`xeab z%6jUwt|u0|i~xut;&yA>mM}`?*l5mo-KWy!EbG*Yy-gNXwoJvIp_*ZfC83M+_DHm6 zWFYTQXMyGsf2`CuF2+cEr1ulm(v5k>lj{AzaVK47R>iai-$-U5q>p{``y zGMP9G3%1DV;+}gJtY5(p^h^3;4Vd?oo`ng`UJa(whZ6qE@@JKgdzilz;=Ff+EqbRF zh*-_obshXkB{9dTq|alc^d3;fh@jm}CfPE50ki>Rf%5f4&@Jyu$`6omqfzS}2jc@i zF>@wm`@R4H3pMpzVPHn&%d+qgs7Ne_fOI2nFM`GtzAd?14Ye#vp(5Y3i0R#o6DccR z@EAsB#4|b6F-tXAlM(~nNE;z%(p};So}0xa9yXt?Cws5PzN+LA?>uKC5aVLU*^Apl z3FITdBP1egcL*~rbx=wVs6n-UyLfR@!u6G#%OwZgh+Yktyw=@XMJ?u9LQ7~Dj}MAK z0CYl+*tTM!KG~~hOCGCQjRAB#e$KVs#zl;FssR1*fbJ zsp!Boq3o<4@c)qq2X1h^lfy-*-O&l|g%xyK1oY$=^Nugz+-n=OuS5g~2 zqoJB|hw?Rt6ZpN98ra2epvUtVE3Q`r6qdPMo7{j1$|UGidjg!nzQ~N)r$l8`w_T&T z7{u1nD&&FYX*JaF3>-Jn#tL42(%)SoHECes&-^3A+>P3Ubw?UuYY+=Ks) z+w?nJcV{IzF7j@Dt`%o)KWZSad-VFp%V{`{B>{LdYGC*a-=9qE5#QJKI6YismEQ>B zu88eb`4aC?%N^cvnE{7Jzk2=eTB*yw>o$`-`yIbavv7k@?PC|^EisUrhz9hd zR2@6GhQjhm6SKtYSEtu~4=&#ESV|q;PqcII)L8<#1tojf1>8PQ{1ke22z3J3F4d@w zIo-SLquJ(o3HN&x-6noe-hod-F52+{Bk<>$64mS>HBn(WqD$70&#Py0O-OQ^_9?q9 z2}!Z-+Uq&hlK5`<>wbrQSJdRaCxGXKJnM%hBnrdqMWs`%;CD9>&+q8kI50}MGp;;g z&XV`x#IBF5nni0a21d!g>iPgVaOyou2<_XQzta=0@2~ULs>w3z?7J#Akg&BkCC9}O z2Nec`UwKQJRinA_*e|h%UVEc7Gy7HOY+dX=Sc`JRj+JHhIa?bFDcfPe7jFOEX+|ob z&m5h;L)JNF7Rg%7j@)bmOyddn#pK@t1VAW~0S#rkj%}6HF$mQX?@kttJ>8@`3&&+~ z+l+iYM63Ab#EaY2Zc1y{xa7z1jDn6-@^mw-jHHy9THtw9+dNg|$LuC*e5mOkaI*)0 zGwA-->?hM_yUkq;RlNzOh<~l#0qOMjwt-P0rWXjvmDW9{jx7M6Xs2ch`#FZoKpD5$ zLxaEAwjvNgN%i1#M28rhr0sc6+vgd_F{!U@bq!9n9C!tMHe-ch>ui0$QnXW-=z@_O9}k-)sZsM5SntrD z0Y|bdA|~&`_8@PIYDwjJ`E-nzpz#dS?z(`pzEhvn==BQQsXFf(m$TXC5XhAGe&+XS zGGebxTJj+fff@r;JL(u8+uJ{1F4!M`z#*=1%eY`RyhihTRDOTLOutItmfukaCmH{) z3@*ihLF?u6%6G(RCv%t9_FAXTHpA|KQ@VQZq6j@Rz*BR3ZMjy^QJu;h&88q!1CprO zXGvk#ubs_(%FfDprr-);i8Ol*!jlzUg(4h|>ep0ojpXY& za?Bc2IN&RH5@hmPmhLnOeiDEjbxJ2s>?c~#EK+K)T`Yhx6thN0nreWgc}wqZ=NP3= z+5`pn@#+(Oib4bm5PeCPe_-4;jxRj7Uq(x0AVRy&<*` zXIu(>*quUDVKtq*0Scw*>Jhd2>M#3X5*Ue_uzt#!aw~wktK#fwHX%Eu8I?!C13==I7ZT#H z68~vrOW0I3#iAR-?-7x;60?TqXr_kaO$KEFd&+gTvfgWuj5+5>ycMG>S@Q7(%cg0$ z5-x6I0j3wC0^yo+AFG3ksxmz()H^grYwx{OAz5weTFD%Fr6S4C3oZ`dSAi;pzzQPt zqWPS4@(y(gAB}eS7H}`b4)5vGd%ZzR5Qk~=`Z-3HPR;lQ#;maEeHq|wt&RsqtjPT{gOp&jkzDeg%w3xQDvbc`#m)P{ z7uS(~y~U_gRA20Z_fsq6B!N#-?A$yfPqe)@exvy@=Y+k_0;l_HJO#l_y&mO*Jdys! zI2W`*@mE~~;9?sx?lUy&a5cMcqkKR_kSly2^$wMPhqsWj|M`+e19PM$vUde0H$*Ve z)RrYJAQ-4s0t6%9?hCwpHKweZPl41)l>4|dcDD=e;@rvJ2n)uCYWJ^qd1z%0;U#a3 zC>%S-X9)A5YLjb}zZ`!_s=b@drHswKOnO-i_@w*+!>4a3UkB|^qI`9gAC;%|6l6yO z7L#2ynn~pFSL6y{rGN{{vSRA&bYCqhn+eAv4Qc+%D|!2eyU|9*v)uGs|7yI>iQN?x zH!lBFB^dR_XfYg}j!vD$I$?I~(CB`=4oz6Lq`;$Nru}pz-lQxzzl=O)%(};KBZ)k# z+6gq+G=i^;&%R9fY%*pYq)4+<64th{X7wr=UwT@FE$hk z7*bM|yqJ^^b+xMeRfJ&?y!=16fg~Yggq5GOWA~^}a$uC^b{1fXP>hUI5c+7Wtd8jO zofNzWDluq_63r{NuIMzG_9n3k^@ROy3%Oj4V{V%OzD@cWq_{(3`!fjdEu(@1+k>PC za)!&EUKH8>h(fTwc5XC%&JZbA{VPJj!ouRCKESm8cAkjfZf_7Rl!nvf>%gviO|AqEk~j{8nd%m}jw#izb8%U%vzvXQTtgxif%=R>+p5 zXf5aQ?YCX3)%0+SjxztiXGJ!ud|(YE?_)_datBH7$^uJEOgY~pPV*LIOWa>1jEZny zUeKe68^2p-Z@jd73o@)wD*H5wsb!&j?J+8^&qlh@Tpy60GY3RwkcJUNh6M*&6|-A# zzR=6JzqI@WR!n$8EWFe}km8q33nd*AouvXJ*L4F5g9$pAwLQ7iyB{PHjk6h%%KiP~ zxDY|Bvy~*b*&wIYqE-=~l&9B(n|4mY72OMRUPaj(60T(9)+Czs24@DMhum@pPDizx zhFXhM?XB2>$rXU|+B5BT!neYqyq4_K=1kmgx2lVF=H!^?dI#nFf7y>xT!8(!Rixek ztdTBs#+PG=l~oF3G|(qkJ97k;N5!uNq)ZgTBTYRwb@c;GHF_u8i#021oNIEwTUS`v zxk@6fprYq{*@r1tPO|J+l;>)!DoRM^e;~Iq&}|*R?h*s;)Q>gO9sBmN6s5O=!S*!@ zjs-DC2_F#%jey zA6wqkjQGk&C$DD2+u!spo*GF?Ra3#1vH<}enmC51mVWvh-)yr}HqZl2)>85r2Z8pR zvK7XP&&uh{^KK&WheG!e+|d3JN8}7`x-?J%8My84?q0?gJB^se;?!YJaXF2^BjXH` zb9s(d49?_nYS^*?1P(+zYMlIob5_u1RQ=4`#Ka#ki;4CbZ{cIgn3{tr*s^YeQFF=R4{w zq|5u;o^$mRRgUJ^IcM*j|udl-{6OvgJMt9CH?{jGto5UO^ zxArvz5SH!#SeG*2oNXR0AFFn@!qzD=Jq;n_>@TOcQ@L|`tj?Bcqmb+*k}rN`rBMXE z;jy{B;W9W=KkrOUM+XZE3W^zB*>I?>y=F%}DgB!q$qPoB_;w#0s%BmpfbikWy;^Tm zQ5hfop`VJLCH4)*fBf<1?g3+#C2mJvVs8ZgxDNhx`~Bhv|9`+o3Y*5Y)&^Tz%>dQ33p%b_V7?TRXz77WuUS0jSC0X2ctXa!)aF6U3B=HuFc1~B(nPXQ23=otnR^|d4wpZOV(R$b zm7C83VsWp;Z;Y^S2*=d6HFTYfEArfiJt^0vQ)$ogj*;Y?m?ORYn=vV158QIq+9@hN zg>^@kM_?77Ri<7KMiwkADLzLKk&Pc0A%i&@I|p-N)OByM+t4$`LXW{*i{0hH9T4pf zp-Zc{$XvaVReN@N{IotI#`oYU(dpaw>vK?M-(#+w*~R*7PFF;*O5ba{BQ{ma3$?S_ zD1XVT4o5U>diHn9OcNJn<5qHW_8y0k1WMNOBN@Iqzug1FTa60Ss91iaU%d%|=cKX0 z>{;D24RMXZl-L=rg@aBpW;Cl$!8+(D3$SNk`l8B(lOG~TMh<#)OA2adTX#y#UqUL* zPQU8y1PUhQuhN9|7qbnmYVxbk+fdG%YEUs_Xg>g)W#b@ny!J4GiW^P6TP4Udxo zR*$W<1bjfMChtw4$77YXJ;SnypTv${7MDpDiQV(I$6S~09-$D5^xEw+aUyd`qYolCs18;_2QIEYDqVNCX4%>sZtGYqjMr zHwXF(6Jn1PG4v5dAS-?d`NMxbV+SuUDuaArqQO}_UX3-lcO)`Q)k~b{WNKNWm=QD zp{E;j7(Q-oUK05F`)Hf^KD!zZL2(yPOha94vP7)roq@ySq;a)0;_HgY$o&D!ArLJ4 zFyzFB|v>`;_JOhHdn=_*UK0s}Y!4}^v(r7>r7u?s|M~+@L#{ugwP_4E#c$wvT{LECt7>D)Ur2~1r86T{Fm0>~ zLE^}x)hL!OQ;a3c&MM|Vkf8eID)wm($;~ z$T|@L&{%vfd{-}wXjTeq^l@|td(T;S%r&Xh^P!zhL??IN9>!TmyFoHPaBue4W6vk% z3SmA)vJ3Nv!pU5Y>6ER1*$O-Mjx$;>8Luf4*{rcWVHeKpwPR_1S|M=Vsp7ime}a5uSX$`bq+4>0Liv zTfbn3?vYl8Pkb1u-o%dg`;9pO@*(7aEY4{IC3d#{F))3FNT= zhc8EW>A~gy8Q1}CS0Uhbjm>=usOnKU^1o>$V%=ySvhz>?E9>aTr!{i|;^7Y&jQfH` zqdx=uQ~u7wQt7?ka~628H!%dJ-3%=))HN;@EzZ0KF`-|Zy|fe#jMKGw20iW&G)%J* zhdHd`4Q5pj(uA*kP5hW4fuXv%jZ7PF-Fj_;{dtwAxlK-6Ls>*QRvB=%EL@#Q_I7`{$O@+X<`^YP^e< z(#P@|3Kll8$_s77J-3L7m@T>$x=1>OiIIiR6wPL~O4-wEYTMEV?Y{u}N!%WsNz!y& za3R>34<_=2J9xKXW*b~~Fx-eZLj)*Fkf*)?j%+SoZrVx4`6O1W+?rAF(~h)<&&q%q>g^7gHF=(%3MqvSwm3?>OwD-sZM1Ij zQ;9m#glz_x1-n4T3srsI!c_!p!9=pEEr z7A1Q?I_4kg-ffY>)t(UBL*B*L*F8P1S{FU!(xz^Cn;4oIad?@`b{cy*aW-2!cVCm7 zNT?Y@Q*{`uSGqQteY7rlvbr5uzI^@6Gzr{tDMxdFJXz(8MbsC?EBhQ@&~`;I*mxE^ zB8-78X{^7JeYzdF>+aaf?`eTl6HcJz0ttujuX&+w+}X?#wxQMq_^oTi-ANv{nvL)b zfUB}?sYDpR_uUGx@Mj1d`yz2ySrYWmXDfyEDFt;$%SYJp1#BerB*Wzxm;5`J`%+Ck zD3dBXXVZZXkL%BY4d87;nW=6`Buaeyt5rXjlr+_`*xV$mpO{sn03SXy}0h+|(oC zb|9&wLS(LUPIltX=tJ%5iiR&6@kurwHYkcE1BHDz6!6?IN!!ZMlGP~vuGf^O%lN$l zhY6DRo15dYp^E&Uy&B9DTQniT*Bd`|XDQeYo~FfKQ$2g5ThlG+C0PQdHlyoY?xq-5 z_O&|O>ln}BuVyDSQyOWa1rx@yEvA0+IFYQrYmbr0a#Yn$+Z5B%%ox+CzMZjxtN9!U zD^Zir9g=8*K@XmyS|JvK(&UXp;GKV|V2j4n-wb})TeX;?@Y@5ZWamMG%aOq~E7ibS zj|NVo%+%C-M?g7l7t8~8pBeXDu5~E-l7tVG@YVckb` zW-9Ug;{Q^DFe!w(ydK1QI4|~dtMqQd`9*Yyr-Z^{_aR4e|Cz`ZiLFcy?;RI~eKLGY z5JtfS(FDtm=aG;rfhwsQKT~O|x%tRY-72Tv1^0 z^D=8ZjqggrY=d`1x-}#WBr>Apv2gpu^;dZs3a;TBBLl1~-x8X$_%w*2{X*0g4|{Og z$WZNt;p^fX8;XSDPLY2RpFSiT@>L zhhF%-=a@+#}=)=ZlSFMNqA~|+nkF&M&11mV- zr)KGB*PB=1LqWXny`qOgjj%x)?ox*#5{c%ca$}+>=Vw`o9`8~oysieS2sE!``+yG$ zc)dGk1~}r^gzfAb-)Xl6LU~v#g1{426lzM4Nyle(ADJNv#>E@nrGARk<-9zi@>9Ep zwmFpszI%Nf`NNV=xjZMtwCn8X0EIJK__62@!voU5JxvDccv5vX^%Fs_YeS{U4^}f+ z&NZ5ceYwFo@h?O7+es!y38lyXrl2s7i#l_O8@Scwhc;s-=-4LLR|07%2w?d&_r$wf zF0q7y@z&2Zsf?hWj*WQIEK->TnJf4S`Vnj7`2nzre1i@!h!oc-8hK`^$dUKWbVuGD z*)J|Q_>G>2bAYfiROHM*T2+fxSg@rGK-Ngp#|U=E9LbrUPPpWgNOZ??_rw}jq{8r_ zLDLcPW2p?~`N`oU1dz2okiB@{`iq^wdfi3_P@t|$I$qn8bfB9Tn&AcEt7`LF_|O;k zwAb-oMJNIzsU@w>Ux;=O3dn)_G_U(#yA8b`e`b{@pK>0`)q@l{KRz#lr%B22EpZz` zBo318q7S_s98zsP>?bD2=bB`WW7ZxZg*b_V`zu*O4Py`xFxCV_6!tDaH&W$m7PBeL ztf}yDy!E}#XS_qmfD2;cMtUG335=_Ugh3fHmeOe@IaebeIK{j#tEwpKN%$2~j;WPyfcq#~QpD z$p3|+GYKY!mR&awzI|5z^_=D32%ea#A4lUypY_+lB!4UNbS6-aUX(IP?rFlRHHic= zznd7GvezI3a=^G!*2=@(L6)1~9u%rm?Co4a+b=W-yif+<+IW~TT2=(zO#-y&v`o)0 zcmhDiVRjHjRThmSdac1HwH+ouWk;d7WO*a_Rz`p%jL{2EqeNOtVSgt45+g5nhdFDM z4N6h!U}~0ZLCf#)sCoLY)oKmyi7RDR>I?e~j+5*AefACJ#egXF-p@oqv^Isl9(yW) zoCr2+MV`{$IoY2!c(YxD56s@ICgSR8FV0?xbBq!(EdEAH|U$B@W>xi-}G|M+_A zu&C4i{aZyqRAd)aQczMFiJ=D&0cjHb~g z?%vP6_wMukV-G#p!*%B~GuQQw^L(9XrbcH!eW$@^V1WgO+Bg8W#1wP_hx;L(uqi&j zgpI$Jav4aK@;_05zdgo)Y^x|Bu=!{6aj80)F6XvS_H!%LJ3quU-v7XmWrbI+prX(r z`aC+AjV;V74Jd~15~wv&LiHG66u+M93sQW7raE}#zPmst1kC>G#axg6~Rs(yOL^UM{Wy(Zjq>+Pw^PF z6Gc&lc^AF;yI?q=#GUwrS_jroGkwx2J{k zL%LZ(c(+67#8VjWiTnXtT&Gr8xkB#K)qYb)oIXUo6o2x!`2tnUO}wYCYsJHzPmp0a;m|pbCeEcBNCyL}r7Z@>Z{eN*WCEf<~eSpi5dN$<%u1Yx3hu z1ye#hQU51iDW-%vjdlId+?i$z~b;p-a(}0q-ZoOLO}G7UAPX zAifYqua!1eHt!>U@U}Wrs!)KMKykZ3=?Oe8j{7u8j$i4E${#egf9`}Xz2=v(WS_FS zop)&dW#sRAAaT3-echLeGjSpMb~B6Ki%e&d06_M}ACb=wP=4T*znxH8Usix9R?7X9 zt1!2p&i^pd&dl$xY(|j0&;EVMgY*W6s^t5(+~4mGDJ_5ex@uy07`c^}sL}khrk)lR zxt)_chXUX|+4cLD(Fz9%JmCp8y0^x*wZJ7^a5Fjepmb&@LyJU`y98g_d5lG`qH+_g zt{^(K-8PX2Uot#7H%9WJEWK#jPRO#*aOAan;N5A2-(FH-kbiz@W^MOciLn8IApZxr za;ZT<#R{#D^OuMJ(xrsgSZldWHrNo7>B=7j%-$47K3(qH&UwMB?e`h4SOYUTMR7ml z!QhRB$Ps_#)?+QQ+rfC6Dv0BfwQ?nVk&yiRRi{5CU^ z*>?vyLLTLhc;Bf-@a`3eiGOKZ#a_ei(jQR0?3}k_Po=C)`KDEx;u%Ocb<%YGs-W#O z)F3#$NlS*%sU>>SO-o+jijyDEmh++jbBg>AX-ix=Le308E&eN#5sbb<(;HBpRTTw) zYBh+#tGPr1-5G1Q8Y91#E}K)6A-t_Z64qmGbyyB&F^sEz%1UCW2@6tPCRkiia{bKa znFh zg6V#U;hCoK4BGkA9?zWso0crD5I!b-$oiZ!Fw(m9Z6BL4vBr`62b*ZHrzI9915}a4 zZ7YZf-oU4kn_zWLIOvdMkF+RxmMK?LnPjVSacYYtTihH4^tSJn`WkU$@r20|gKQRu z2|E0+E~mjFA8rJ9ZQ)@4G3dY7?HZsIA&v*hNA_1Paj3vjdX5$dR6m(H_9xdmz(v zF}JVfbMVJZmlF3C&~a`ao>7liL*<89umfwMrKuY^cK2T$QKe1D|LU*~WgQ}~TgT&{PTHRfYrGjE0w0|jR^^Y6_P56uhyO`(Q8LsZa>#{k6f z|8buFq&W`c4`KbvD@6jy9FtBdbSS2HQAGTXmJm>Q{G^23Nw&4hL=U64Y#X* zLQ%2N!wbMvB#t=y$5SQ8t} zTHoWe#BSWFu^8)X_yHv;F@M1@cNq9kr_vhkZ19{1HvI5*CXPPc%gq13cOzUDzD*uX zBp`nB*N*>B&ab$W)gF3BIm@o_n6aQMztm>oJi~qFi|Vm~j6_O2w?Keg=FU+7ujLrr zB0Q&k!ITjIgFtdA$4w@X%GT@QW|QFXSUOX@k_*@2dLlG4J*>FmYsGxMRK09VhOMDd z#GR@>!Yg@#sZFksv`*ev7U2#?xyCzTwx(q~gjMp=VbPnmV8uL1_N3SF9*C4#cjV*w zEt$`5=RuVdXW$^p5A+?+&0rAOzCs<^_u6D84Ub^qD&OrfjJ;LlT3bSiKtlH2$W`gt z#p`z=KB#L2qfMdu?)#4mY(_q={4$qF4i@0ib!67>g$P^6MJ8OBndcYn(l z>g`OC*DzTVbBGtUU!gZ)>UJ6z475m=#5Z5MeCD)kwK7DQ3dnCwl(#dVLyQo5t3BA1 z`CTAaQ=-D}0+jKVTc55ZVc@{%yfmbGIr(851wf}!|Al{f(*)=adWM}hGqPQrVF4gf z{m-)h%}F{UdF#UQcynzFd08hkt;VR^;-v?aJ*#2Wucvc&?}&83hqIdPv$1mL8|!tb z7FN${l=%ZFQp`b530ex%dWe#{#dY3%of|xs>}QVqGqDeU8w0O1CEW$GU!ZhS;xMQJFf|Rf3GD=nB!q_ySq>Vq8o)4f_PSO zXbvwWdaUL6=P|x{?2mv8s&tvP>i_wS#J>R#Ma#GO$aLa6WBY)8`NyN9m9Npw6gxvJ zJ0V05`hmKISG5Od_Rv^Uw5pOIE+$PvX8Yi)%&mK}S(7DdJj7gfhu;m?Xl`z`JXUH) zcK#G%-?`ws0cP0u7+{%aie*=L;;Ok}U3XU?0Wn}AoM}c(hzXK$m;!bG+PXf6dikRl zIcn?6sagmS8Y>)O&0TUdus=x;C)(YLwX zoY@|$PJC?$j(>U!USCg;4>jzq5Qir$cX*T8%N6DturF%;MIR-Y65DYW5{$coD9KB2 za8;UL#QYx0zkEv!VrNh1wUfJLJ2Xu*(}7ORyJZ>sM5gMF#Te9UH>=wf3DWqi(6v^| zv63GWUs~TND##m^rt40qDjwm0zJ%vO$)Q=jI<+#mz`lYtce57NhhxK~fZ?=zwPDMP z6a2H42Dm$Jd&JGSXeYW})hsqZ49ERLr~l730;|)nSb55#k#1dKz~l8Zrm^~Und{%` z!`~IXFHUAqGSHvDkAr;M{W|M~?;RKNF&r7-@A7n7hMW0UM}L_>;JWFe4YD2AG++30 zCZ4DG%Nlub2Z4it7;08T(}!qLJhWD-w;oP2k8mS5l|M**MKvC;>&SEOo%|6?#z}8S z0dI39kcuW{sb$=6B;=2$LyEAE1jrKX}Qx$6X9Sty`WWN=?g^rmUZ>)`q0(?J|n zClsga0422CPsPh#vm$VBczwEvb3B#7HtA=Q6cBj~=ZGl**SMxU5&IpM-BnD4(;330 zKVGad5?s^kTk$$QoHJFS`Uh{@%9qeoQ}(^oaZU6lPl-$%3@JN-$16E>dvcbdZ~b?* z7Bi9M$;e!8aGpSH#?uo88utq9GN$TWQ9z*0hgAe%g>ql8vrb+}%p4HCyr zP4onlEG9XYzxZ7N=JS6vG{4N}`&6-Iv*Xl zhYj{}H3woEcNvc7hxFe3Un;IVPY7(&t+2h+HJX0iKYW|K zdJY9d{Hk$x*3w*``i3NGak7&@-_LymO`_G}%l0}{x?tV@`moK0jz|_4(jKX6g;REs z%&yfNrjql6TEq&?K?3#SKIoz!h5>YmiAo+h68JiU!G-)BzFpJbwN38Ok~JD_A813a zM(M<(I67ERpp(i&Vhr-K)}&1#8SQN0fr+1gXFLRAB~0J%)HC&=X!;DbrrEae8{8ztVlf zSAPu6HLqPdA3*ZYO!1xRs-WMe%*orpu%ybh`%1U&hI0KY~e(lHG1Pj1=>F`Ul zw36gm7=7&CC%A%*cv~Bie|J|Ci5V;Fc%8wh8bHv>^hcMD^N$tb+FxHUkd1tYk|ych z^p*c^)Ma>I`7$L(oPqbHzb82vuwW{-)(=0}?RcG%akk<;>-ThX=D0s$f*0jB`swY1 zSBD8VzXPeR0$s2=SP%gXXbR()C14+#dXVYR;R6+K|2DckXq;te5ZPmGxC)6Ky}||4 z?)_abMw=X}(=ehT{a}U|deSO?$W~*4=Ae#QZU-$H=xjckqmyhA-`DX?s@SKz5?FiV z)L7M7`g(|pETg+X#jopXY0XCuyS`JUGuijMw_c&Dbqj0;*ArW9vNwzcyIA*)SWUq% ziX-)pLf-G4`9_iRBz?5VR1%zV6DxgfqV1{hVdTR^3xAoJOy}VD&8zMYf<_$an1ZrB zT9~ZDI2E6?)Gc);}GP}UhfnRUY+I`b>>(sv{syDrtH0Uv=R9(YT* z;-8|@;naAl8Hd7RodUeLa;W$zU4`z0$<*LN@b3YfgqI8def(4IapAAMRIgNbzV*b8%n5RJNX0wpM=NgsBzwwlA%`j_ zHwhznBr*fjDtoD5afNR}vz#{S(@7;wCqbEQ@>3!9=q%>zhu(v|ok`e53ZT5pIDY?$ z;6aE;8l29P{*ugx!WQX(DDG6Buo;(Q;OSxh?X*X4{&YT0;R9Xw{W~Opx_?Bq|er*_+*#dLXn)uze(>@%2cbM6*RJanWk`?IJ34PqYhOr z4`1#wrDZkJe8E1oR69X#o4^FCv?bw0@piXuE4XqOAj_gRQ!9^54i}?aj*m-j=~waC zO0h;*^El+oJEqM9Vc3QUVXbrok~d8CnYv$vHw)=*uwHmj(dJ3m-NUixjB;pH?ZS&l)oWpTZ+JLa)(RxwAyA?* zC$h{R&%jN~j#SCW(a3yw989IEpFvDH9rzgJoO1x2D%?+*pX#*qRU8E;%Ajyu5fqlI znFgE4o2MsD#;=D<`SnTy3l9F@19iO@GUFi#YiPd6~W!~YX*jHU<-McFW8>!ePdW7bnKRRp2>3erD>8Br# z->oz^_QiU?$LU#?5N(!ccuLd5q;Xejif)a9XoIs2GqE&z&W6b0B7-8mFrfS)?(IK% zO9Xp!OJMfRaVcyDU+ho<_sMWF53$(D@rvGhDEGhoA-ZtxYTs% zVRr}I(_;Q!M~MnVE@64FX(2+NpujVP!p01PI%J(AM%;j#{pwh%oR`b zccAUpR> zS8*6YHmLm^5jW4)@ur6{4)Ve%*5A19lmfnRykHAvRM`vhxc2P86S#F$c}v+8i@Yvq zM{JOxUS}fg47wXH3ol?Fm1(A9dC@zeNy7q+l+3nw-sQ#o}J-o z;6N>P85GG`Ngga`M+l!xhZ==|C-3xuhYp36CI7PP)R8-`^bIL-aW##x{DJ9NKLW5l5%43*ea*9l~P?K7P$JA+~^)a zpIekc9G}O%1nkgS;Li*C(QO@4Zn?x3@jN%DD` zmc&apUg>1VeR!l*CSAoBBDKsBiDoju<~?0In2EQmW~dZU9voGFUMHiW;Uf4wdbHQg^l-8AH*gCLgo|JyHN!>3soVFn49$<>ggdb88j~i&{~}@oX+<96yHB3 zgO1InSH(DoM}KGh4N_lnxlj%W)zNGQE)xhhFx&ivc75~kEBnOd zYga5jg}Ne1#FoT}u1{ zl~-&m4ZA=Gzd@4I6{DQ}mwyN0>D>n#rYUn(`z*T!hlEcH1`mcBr6yRfK(pI%W%U=v z>MT1p2(Cz%J23&2;2*Ca3QEBW**5U(RUQmF|M|m9{fQvH;*L4G?|WtASb)RJ<;PiD z29B8S?Q!-^q8~hA;-^60`4&I%nx#-40!L%^AYqsn&bejc4=&k$LS(j?SxY3NG~5&&DW+fQ>%D_1 za9X&)IGys7cy{g4D?mKs;KhXPOTQ5&bC>W;(>f?8bJoMuL#cPkn@FH%N~|_w6_t*T zR;H$#G4kycI6>cy4ZY%PzQ%g!|5$pL4kd$5JRptS=QffE09uyt%89{4uIH>0%aLjC$779 zrM;I+bSJyRG70ED>>Yh0`z0w1eU-|FX7D|~8C$vM@hXAckp_Y(nJ0;hAD5!tLA;%_ zOwbUF7>S+8_3iX8Ol{O3UC#kgdVSd073gL#mUo$!ZFOT%wqhsd(Jt23>DQ|a5sf)62MQRUC9%qUX(EyBOLwlnKfVOrC~jR9-Et@L zI&7uMH0GwDR8|+b<bt$gmVM)Z*t_45sFiw~hQxdBhQT$HGw}mpz>S9M!)1jsd<3ffl*ed=FkGU& zpfBD%8q3^k-wMu@jZd3{Q0Dmm2N>hC*1OZ<<_{9#-Ma|ksdBDo=mw|)#>1dbwhVoAcS5ty)?qX|&EfE0?at5Hb} zh1M^II;5BlGMt)%=Nxk zAh<$<9U%*rE1(&%tVlklgI?y zZ&@kC=o&u=g~Dba~|3ecbhEN-p_4w=rkY?FhvA2w|X)``lTnym9TlDq%Gghu7dUw)1 zX<6Ad7HkILdO(N)PLn0Lh6Vib;ZI|oR>$Qa3 z-r;)~I&EvPB2CCX)$6#iC~_0YW;KKqwI#u`isghd8fg!)7lj89Zb{NN`YVnBrK&G^ zoo@~H(glBF;t>oR5LTGsIl29!L`3R&bD7^S`&u~ydeRSwC|MP+W4 zmvCB&(nJED#>S95&WCTRXL}PR*Z-3*evX|9$p+;0$usygUHUUj^Xd&s;T1%3k@NC` zT1p3vkM0dy!GlT1ijE7+%y}?J{r2Nypj}tk^0_#9K(s0#+PQUGgnyTaZRS_HPqJEQj^cEdIMDtMQExJ{QC znc^BfY2E3yQDlyh)-M~W`9H&+Y*vWHJ|uZq#zay?&Z!zcWxOlfp7{A`Ho$r?Rtyt( zQIkW{l(N8=M+I>Pq*@Fjo|+clT%|76~ui#)w5AM|H&}i2*QqivlxaEqxKFC)=xx;d< zuFkt-K&+tt2zFPO49c!OsxwsCcsYKw{Z<%rxkOx-lOq)k^=p>=d$n&vdV{SUoIb(d z#t|ASPl;I^@uNtVYQK8CR%$`kn{jc}@xrEow70?QYywS_d?r(Ce=6isYHx82#C{8Q1Yz}A8eoqOQ@T2eoO8_ro(C;Wbc&qDle^7kz%3> zXDYeRohjdgM2?n^nRXvcMXx{j^>{hEc$EV8*NMHAk3E+UOib$|mRrkdLGk7I?em72 zVSThCDj-KC^XN!#q6O5s9_!MgS?vPu*XE&s6*C4qaUD+)3(z}`hl?C>kmDuAa_ z-t#`3dsW1-iu9_*n;-Y9P@Q#_?qF@g-9_p7^hDF&lXy17$!3}c2 z82_`|{)I@R1Pc3if1S5af3)*Ah80%NU#~s=*^S0R@zDUzPSCJ$Z|e(ok>ND@>cddX zf)fIe@VN$^ai;=#r7Ct9nv!>kn9LC&Dh&lE&BX_q?6ymJ@p>w>DP197RDWl%ehtC~ z0}qs}+Q}d^o%D^L5`F11h6C=WO6Kc7Cz&SDh4{7AKtB0-5S!I5TdB} z1a_5kC~r|mRQs-B6Os)6LL{?*CR7K)uEDB~fr^D~F@-to^CNp6st{qL{fEmVOdH~R;H3TL>|*9|9L*Lvj4jkx%|5~JmKhu%G86H_I5(NT@y zt5_#Y#8O7u1m9=05TJ=_Mgm|CwCyOjc7$X?sBW9U8Fo;*^0WR zo)q?CY`%=&ZwB2V7Xlw!Nj=T}hQ-XTE=Jfs0z~6fB^?mMAFi}NrRyb>h&O3>i8i;a zuN1{wrZ+d&SCN0Rxc3t0Zan?QII#=}aFA^=s(z+A8>)$luRrS;h#s z^W*^*oPKgyoQ5RvrMYMRDCydnWbht5DgV&!@HV@w@FV_1FYihVuU_MfEYCIR`}ZJ9 zCdYLn+nofNwFABp)QJ=c>E5`nHX=RBQIILt^jA`uIq)odY}#kgIwmIu^ZY_=aI?QTZo33h<00#JnR0D;koza^f@fe&C zihN4w$P|whr>L|3lx_z)_qZ{`?xAgbQqM&0i-%VJ&V^Y8yV}V*%84)c4}f;FC3E^} z)OL=Q3Q#$mkZ~%mC5Tfh4^G#~5hOdrdjRWNbpA%Gzv2%p9#1rHYxi3K=dcShh zLnTLNC*Z-HV`heR8FlbU@C4hnjc{yS$Vt5{UljX>y2Y;> zsxvP|K5rdhh;_aA+69#kBTc1N`0_vmD!l5U6mZF`FDwVtF)B{fAjoBF$6Z{;l{7PL zk`b60fGPoz-OPN<G6u5_s7zAPiyNSTQGL3o&UsIPGh^!QSdm3jg zbLoF0Vq{$3U>B1g0bb~xW0d~i^F}T;1dy&=N%%I5COvyw?v(tj3hZ+_kG#CwE$noX z`Vc^DlGa;nKsH`o3r;E9g7DyUNm%V9H@|D~!0MHmY# zf7GQZq#dFt4_1@EPEjqhL@5xmld!dpm+Hvzg$edEW{*2X7WY^R z+nSEKrum{E<1RMEXCh(~H961t>w5KK(;MIVnGS}@n%Cx(jNDB2zxwo5(%{}KFBAef zdX?btZdyBmP|>vDy$MAY%1`+y$p%~{Zis~6Iy0eSI%*&;&U$=66|bNt&{pMsaKRhj zf$?JRd*vZ6aybRjo69-TE`0Kt3UXf2n0Np;CWxb8wGu;!CS`cty7y(kCU};+P}QY+ zz(sL}0`v>@7{|)v6H^ukopE|t7q2>yj~52yuXvL~_iTRCoF)5mkFM;VwISPL*SBf$ z=_IuG0fKS~w6h%&Hu{IHE3~kiofDdwd?#bbhG$9h*CnSy%={6G@sI&P#04@iyJ4l9 zNBcljRNjH5*t<^Je=xn;u+t}8kqn5@-GE1}6fjbVH2ja|dK)l@aurz^NT8B>cm5c- zzUi0*E3G}w=LN*paWp3YNGG+D;CSl{>_q*jo9{wc)(mzG@Zn6&G<%kWl0Lj)m6H4l)bbG%ia0|Mm2)Iu za*zZZuftpL4kAh@>ykE-CTkv-psCQdU6HYO#^ z)vy;gJ_1B8-`7r|4EGU*7*lZQK2&EO8_n?ZpUlK+Nu6Eab?u$W*JZY6haWsdC+^Lk zeXgWo*0fm(kH31{_$EBAwYkjI405@_semCY#mUXSj7Bn@2t zNHe&o0uhM67KgPq3q$UjgYcoQUh)z|QN+%q(!`%)cU9s1uKZLwyijxSKh{peR^|Mg zqZCC=M$XiYc_1KGxdH}Pt@buDnPRUzBrb0kW}M{TcqI}`DcBbJyrHGcb^n9d>Dybv z8(g)3N3HqGaU9b&%{-2t~Ve%jy@-_2wOQenf{WXAjd(&S+?-LXv` zd^<;j_{O!rg3K@NbKW<9eL5#*Jq`oR%++bntqiccHQ@2tY`QqdP9U&b83}{K7Sdnw zpdg&8=oiA1a(WvNLlr`pYJGgQL4wKg`5q_AmkYLhkMVw`g1Z$r#~tG$ zq6G5Gx?o%uXPvK06`g+bM>v3BSJ|DWsyrW%GtNnYnK%VapJOZe`%=h_Wvbt%b;j*`2@(G|JzgpHom;jQlzSR z+owCx-5X$>ybJQmb%m)^}NE9t#7$yl+DODdBsd8xp!oOE0j&1yyp(m&)CdAu=ME3$Y;f(A-SP1AAo1J z7qoSPD6f_*_-1WDr0NZ49_vx?Rd9UL_kcBmv8AH)iS^m}+|fQdsSKrqshTQj?LZ!}_PJUgFM@%_ zxiF8hrzhOQB6(Ax@nycwIr)&U66>CoZe4C&%gxP5u;pN>k_Gqs5-v+(xYYnMvEnJi zF}PFBWawaRGazrtvX06?@q71`i&>mVaIV9tvm*^tO$@v#xE})P0`t|bcmAR%yqwEHXT-#rW*nq_OF5bjp-u2gCkOMS+bdi5V7z(3m$?jZp&HjSZE zhkxXtZzqH3ryyGZNW`W!vc=RKuK^hvNL$xwvTOAxWLK4tC|@<^=&I~EyPPpwli~BC z+&TH-Ny0X{6=L$4$wKJGo_z$!?Y7~DQD{53x&zhO9FPFmM!Dsz57@?8y=0#Pdmmnh z%2f6Eop1%b+w5iP=ICh=8Ia4=2f@>b;x4N)(%!gg0g3dV(kKq->sd6{Ec?V`;brX~ z$(#~SdXGxpp4%=7)T}Z`mBcZtvr8%9r+cU-_OnYWQSKCay)QQ%2YcID zFa?tuz#g3~GKr-ag$Visuv?~RC0PB}g2KbX(D1RmTg~jd8}%Mqv*8X3$x zrNMU4#wt#4#N}Q8s6J?2p)S~zIC8UulnNIybb16f1#>*6@3dhG%qPq+A5bFhTK-!; zK%KFAw&mk9nnzT731By;{FH&ta(8T-Svy_$>*#pZ<4&~}%>ox2=HUpb6;2Bnb-#7U zoN@QG4cfNhUQj{U9gr=3Grkj|T`O-CJMpdmzD}(xvR#lNC{M%{-|xX>R}XdM^QV@S zfO9$c3cL-1m5=4&32^y>a;<7^MWB1Ju$E~(gPOf9QPO%q*jW%%Z~EE+qo4Kx+3BOB z*R)kk;*XAPT@m)aWieW11^8!xCBRoX()mrIiBwo1Di-C537P@I_fNY9vjuZJaAJT( z4ByroJ`ZifaB8obitxY+b!)=01Y>MqA@9P<-iU#nTY@qfmMD$FVy*}DJuyol;POWP zxXpL4qJI>KA-!+@HWeJhqF(|$#7QdtSPNb8_)}p4Olbw}_?O6NzD%9n0KZ(?lO)s! zW3_(lq~aGZd}{-2KK->AZ{D&!`by;eh5Yf}$vGGkg@y>sfBZa5;e=|e2Sh@moBYF8 z+7$rrYTCB!fB7P$2x9kr#Ex^ERWxs?0C+N#DWPUeur;i}^}`^@wx6)`Vsns-Lus_5 zfu+^KdV*Xx<2e&JvX%vrN-btPCl%afuE^ACJ)0fcjA2fI^UW%23Nw}V?JSY%MfOyy zd)kb;xi^RTIvvn-OjbE<^|P@am|)5yAmvU~Kk)HQLWcPW*ydC(w`Qp>xRL?O_xWTx z7v{OVD_AmjXIP%m5<)f{yet@(^>}v8$)Iq6OR>VxNXy8evDuTDi7KlDPJIxOCU*Lo zf}A{s%MIvRWMcF^EV@lzEXrln(XB0c&gJp;KI{OH=!`2jlUdBKy!&Smcm+0X!jV48 zR`<6TkL3@N{(c}kxwleI1JyIe3QUIpUQtS?%64eepw@}G0eq2$6jE~3=p+JlYC!D4 z*n`4HtC@~(^B>Id4B!$ZPx}x}DpA@1Cv$GmiWz6V6^yCgNqwQt5EVU1`+FfJvy&k( z_LD#YpNbl{n3)6ZHZs92+1hp>0;2-4NN~X$Y2TVZs|7|u*Osg?ssLcdaBymt zrN5ozaJJvVIYG}gDVR?*`8+HS&%TrXPfwpl(LjT*W zMT;4wKLT86Yd|eKM;qH``Er^u1I|a@!^X|`Zohm^)$IKAXcyG8;c9aLCR7Ep24|wv zgiX}|tNxGcpRU7jmlqf52?+^%Dbnu%CTbSJ8|q7rPo$nz4T8_>FDm}PJ>c1C04d=I zLw2OBmfnK+r~d`rzD7ai1NLu1H5I}amu8oAH$YkGJEsahlo4uAn${M89e!^@UV|Tc z<0H7j;0j|~wv-Uc`x(?MAl8ndmmDQvgsCXDvqf*R%%SRxA;LY&AB|eG+?oo4-qvb; zTzMZ;kfM`PK-r{Ladumx3H4hB{5iki_z;_5;YKBc-4@BT$YX>Ew_@I{O;=qUVF=76 z0oY9Lf#|#4ZYrC69mA6>QKdtL9Fs#fT(GSU@&oG(cjdCVvZ)}cR`2vyu>PHUmtLL2 z*!GiGJ28>cIOke+Z34G}bKYCI3K7T>uoa4)C`$FNV z$oXa5T7u6yv=iiB)?n!4x3?WMXPoP`z(|hyvOZ^KwYWd- z%|D&>HE0}Elqpj3u5~NC*gE#qGKmf;z!W@)A;S^WQsNKMs<^W63nse^JwK}#oQ)*I zlZ*6hS?736Egh`Y%RbTOAOZ;Nrsr=rEJitY5-LUL6ny8yP)^28vvO2sfT>!jqtwW^ z5kG`qt-mr@Q7w6=>+!g#iMcKbW!RAv8tBroMD4e~iy^(WxcDPSD*ouPd-%uYXH^UE zxgFKybJHu6=SG`>7h_ z84DjUcxEXo8v8q=e@1WC9{DL&7xehp5`IQjFIU*--dhOcWs{4D*P}2i+nmgd@Ad4J zHaM!|9UO}rg()3w(2pSk011LlcwMI_CPzD`feG>~$M`U^8x7PpuOqg{IFQ+x;m?`{ zYJ?U;m_Z|O`BgS(rVx4z$fVUinu~axl{l!)WDKI*sZHDpSC{SJe&e@(3gjBKu0&RE z8{Hp&K$_IM#*_>T*9S{hWOMT+=NnBRSF-$Y5~VDbh=u7lh%-DOOb*6@{rTFrDZr3M zNko>J#gbos$lT}fFCA9%gc{S?4QI%N0N>5`o3S)T1>36c7*_crBi6$5ON9#0BqUT7@sP^O?GsIZRSnl^gBCB ztltBak;B9b1_`;=EM5=X0=utfM({^>hAdi`Y9KHWdqd13p>d?H zGsuRu`V3FGdZkOt=AdK*2#}XMx%V9I)l*s3`-HYl*nwyA^Hw;ou%hSzFz-cfZ%hGl zE%v~-oyYd^=ROB@Gm6h!yPCBXclm5%fRBK_!H{M1?mw4@ps!?s${0S9dyXOApbb!_ zl0f_~IVOF|@iIi-{`x%>fA%J>A3!aL&H)erA@G5msBj_>qs>+Bl8xslApUxP9oYsl zWdp2zHg8)j4w8sEivq&+923~sDGt2s+xQaP&_8WX>pYDV{dH;5DrU-lz>bBr7+P`l zCKhA&gk_5dU&$YoKFlhxjHXfrBc+VWdIJX@yt0V0sxpNg=XaBLOMBg8ZTi=p^Ja}& zvcPm#5?SFH%aYwJ1|MmAD;Jd^ z7;SHuQkZPC^+uRSHkN-T8)u{%&s#+?QB+4cQ*oeD*d#M0pvhYv5L#iPzmHQUIBy|; z@S9cqgEARNSfs;FNIDkn6LIPT<$C9D_`_s2VLDSITE5p6O|ajrYJ1&ojx$;}(I9tk2Bo9Vz@teLbI#_=`zH;n5C#%F5 z8fV@XmrjBcCZN32vwgJ0=>u1S9vRnyw-9~0gJ2q}*(mMKR>|qtnK{;w%etolqdYtU zo;S6N#NPDDdHv}Q0{-r3c9eE!XmBvO?v%&?zumO%#h3fkp{Sor-4>RXpY}J#R1I7< zCu?{`L==DvdO$y7?rSx0mc}I|WeyAs9GCt|ngj+>=?s;CpC)tVoC@T_!O0QHo*=bd zjcs`R_)+~Wkb?oNyXsA$|H5?6n)8hZ!yc=HA3LjTV6ma&<1Lzae(tSZ{POL-c8xCG zTPL9V1N?KS)A8}KIfzyVNtU)pNiy%-MU~enm9ZZP*hMTA53}`4AKqsKU5Lg(P5d+0e=QeYUyg1 zg1k%{&mS&U!}>H#4qjULJeb~gf>7LFzck&NDQcYvtcnLIfqFgIKUOc=(@uc6kwha@ zPbF4G#Z-&UxufAhO{cOk$cRdseXXL_PGfXNmL+3&g^=jWW)`Yq01+5yoe99%a(A z0#{Z7#J#Gew@+6V9#t852+Jy~ygyHgn~UIdF(zqle*JjN523{S$Y|lX{n1vq`Z7AH zw>{Ao>~3;m(2((N#`pFy!73_5cZi|x3ajmOq81^9J^s8(BLvAmMRh>TIjDpp=+2t0 zLj-J;Z9(9Km)-M?yRFB}fYe;A$03}jB>~z7BiEN9WkSh3fbVXlJ=|NQRK>vA>NvvL znl+ft#gpY8;^AE4JLb9PIsIV_>7`NYFq}AtT0g6m`|bfp7wgMOI%I)bAoIQbjB&$@ z2;dBlQ`M7xv}UU)a>5uc20H`Vq9<6AIf#WhjxbH1v}-ymCsOGZz{XoCc<+}bR}A%? z4am?`D4Z|8{YvACLAR*-@I=MN5c zo2gRa2=n=&RCMi+56Zp8A*S6w@n?FsKd51UGvB*S8V`Q+LC-Xny_?2GP%m1nnLAJc zq7yCld7g%;d(FWA1$(!V@vpRA!^!KuZlu#0;@%Ug8R_v$E zhnzTl|5SqM;;HDNuToUI>@&W1J_+V_d`hRezX#>(G^o+h=MG-;7R~dL9klZMwD6=z zrxvf_1)Z93Nby9>lLqN*7i?PJczhx+z80OE?mIltj<1M>HRanuhv(hnKkYCqRm{k@ zT_L=+>*7_V1NpS4SCpbVlSqv}OIGaKN3!=LrAfMo24~+exW5FcZ$MPzBN4osNHJiU z@EsxfoX(?4`o%q&Sg+07ONj8-N!Qe*`8{1DlX$=En>{#dB^WPDQ?py2&Y6oius=|m zlD~*Q;S+Ufk5H3a$`*r)S8WQu(dU1SSD<)r?)zYvLYrbQA;!|MJ##PkMrw3nB?R{K zxYYdvArEqQWSt;cg~<0ZcRJ7(rVCo=m-CoGBJMI$T2MI~(MB!j-VZ;ITlvV#Tm^x} zHEEgvwGaO=r(Kx%+6TctaNE65%G9pR)hbOov^u;!UOVbC*4ak_1+)W@Z!~?2uN7if z$I{k2&hC88Ditngww$IXXD3Y7?hC++AKJHEKATEg0P$GN$=-KH*ryvDLQ01T4i3*% z4vjRJTUTV664mUC?00(kQJ}O(I=H*zn)gp)vx|UJ0tr3Yki-Utih9|W;kacTxHquP z4Eq$#B;UrHRGA)H7u3UNTh~pqeKGv$Asg$;(--UgmQ%GBTC52itU7^0N0jOvkrO8B zLns%6`Acl;Q!eJit>{wQq>(L?f{!hw-kb7jL#oa|#t0SGx%`W4$m-pWrFHpUUYtw^a(4=_)?rE0J zvP{uG*25p-K)-q>82*V5I?5JB!5%GP_irU3IO>Y3LDwZPjhtMFw304LxxugZ^p2&? z6StI85ifg$8Mmi1hmOW7SAIIHj+|yN%Z-tFyLtU=dBZ}oc5`L^q_w!OjB+Z4C*|M+_As3_a^UDyB>P!K6;0TqUn z?!ll1M!G>7q+#d*6h%^yZbouwq#LBATe@?Ip_zdpz8jxse`~+{_wM}{Yq=J<)^*?a zb)M&O9`QD>f})8-i$oZH0pJ8rv3c(SXMOW_fT?v>lOK0H+!ONNX}H?nBb4faB7jvH z0D|A6D~nx)Un@D?$5+Q9v*K9O(&Cr5UO&9W6v>iQz}o$w`V&EVb(&@rn=-fELS&QX ztDrhfF2CT9x9}MK?lF4eD&PC@(fC&6ce&?1+q?~2SL^wxO_4FxS$%c=S?^lxg*e1Q zWDJp~l(L(wsy=JqK+i8L-<)LMsO@oeQq`1_NSyj2I7`dvl&=}VMwV+-sFS3^eY4{3 z{&%7_N$^9GnwMfLx49z-+TIvAz*(*ygP;H$c6aUF$IF{9d~YNh-YAl|VX4SkLlLNi zpz08pTi+9osodGNlHSg3SuIMz*l)U^hf0b_6zUj7JbVQ($*$oNZdM*r5k=tOg(w}! z9w^XQn?0-J4e5N$TYCc{(W09n>X}$ zD9eeH!G?TVNerx=ya+`V@BCDR>W7MLa+{LG0t!;)Iski&^ zxeG;L!8`g7(lV){hJaj?eqE89KcS%6*n_XNq_noqiLV3M z&`g@@3)ebEU?Cs^6uON-rm!F8Uq(QRii)cvtU=Wadmw#Ij59ggqy5z}QH=2)K1b^zroe!uc(Si<>X4Rk|8J8jwCi5A-!FDQ1yY zMhvG+2aHrFk$L4goL(iIn-87vKk5<&4o$p>)_Rx5aPq*|8-FB05P$z?86o_Gm~n}k zddh)~Xdk68Os4?8F_-I%3Dt4HkfvPP+yldXEF)2M`%i2yP97?L19H~*shC78njS4l zg$uIXU`6ibl9Y|R+L2&w#%RH83dq!Me$^V`w{aTv!Gv&Q`iiAJO5%RA8HvzM{jl;u zv%d1I{>3u<*K+*N_jCTld2=l6XagDGHf~e#+!K z)IlP`pQsSxmGZLd zdsg~;rxQmeY962Tv>JQN|w9BRE$=yjySWS znz-%exWh5`rsdOmb@w;>b;rnyblMB{rU{qy-j6MoB{aE>3Bxpm49{>h|HyYpjjr65 zD_MEUe>*G-YL9cWI}ZVv@kAaC2G_GkE}7%$>r*s}bolsREI}tpqX}|UQ^EC4MUG>P z=wIaFX-xQHB;N5D`kVp12GLYMp?7bsZ-o%M< zamDb-BRR8wp6Dte!6S>jQT;3ap7l*m&abmTv+dQBj+LVbD3`t_m`?p+TjvD%Xq={m z39)I7t-5-zZfi8kZiEnS%7S6{XGtbqf2@rI_i&64VHsUb3yR2G^|9g4sxx4bQ_q+> z)F=#VWjF2>8-rpBO0jlqDb67h&`8%d7rrN_x8;-u&t+H~mcI$A@FZlG^^1h8^(MS2 z4D1|__1%?c(x?CDKz>wM9b+Yn-(q?06aFhM|EN=HKM$@2D2G-6;!1k|4l;|&8+5Is zab#~dW{mhT;Cp#k4vb^?IJe57NR?~mXUu3~`$VI+Xaj)zWKWh^sCs#cHUPn(2p~ey zKk-ulGkrEem7y%X&tfa|s_i1!dezcg?X>4=eU)OTS{5{}Adzzy_tl4qudU1kCTxrI zxwPuUT`_M+#S@6J8>+SA^M!@RRoqESB~_cejUqL?~gZK#`s(w!bg_FB@C)W zXoMVI5~~^jX-eI#Zr9?fYq8%^8j*J(K>yd=KLJEmDZZDdM>PZ9VJ(Xy;Sv+SLR|oy{`+yl_mFP$#{c{y*FP2OpBn#f!N)}h z9fT7X@BOg81)9};^dI*xA2UF;$!E-2nO{|#AYBEVOMU=rp%EzxeMv5_$)1gCOZco_ z9T(x>*DK_y3Tui6z_THPIo16&vG(P4$T5IHoUI8q$q@lW390ETV4jo*9y0~4$4bOe zFYf@$EzL_WfG*GCr^$44-7HRDdU<*ITFvDN#3>*FiJj(UWktmhnWo+=Q8!ek^cPxD z!3TEfd^-RWg=nJhSfOcyQtdGmP(;oK zGjp)h!DYdt00{R3XC{VP#wjw!5Wg<9x{(=%li_yhi;1L*!Ab7v|2c> z|1p-t>0EzbCo><;(fcVKGgjJaL6*!t7O&qBK1jw7n&-&eh*WTgik~R&H^f_ z`2Vb=--u8oTY}O!w~6G{wo06y@W;Qod3N{L`Ar9(o7I<=N+4^TEK!{r^%<`AXT+av&F3m ztG~)63d`%4_i2J(+dg6SplBFiHIsut1c0%V18Z^gUfU%{VQ%$rzixE;-XuNPWpa_o9&g4SAZ!lP6X*5R6yzEDFH>ISz&pjsQwv|W`ri) z-)}L7Hf`+92Lch^>JVZ0v>S%6h@#V2#PtWrSmiK(53p7Wo0^`sZkUeEm>7u{X||C5 ze*8I4HJsV>mE-^X45_GRnE%*^y8F|6FYHa$z9+=ppucf=_tz=z!db?tJCM$dj5i5W z9*)odIT0llyAA6ijN3NoGX8zNl;$s-27cTP?pzLzk8DRUUhhw1nyy8Xw)Vcp#(uzM z-o)~HShMMDtaji!)PW^kNs0a{-1nTRNzdiA_gKC`mXN22KEiV!=&j5~s^jE$8*M$4 z3h778m?m<^c-gCK6O3%#IsVL8--Rk*4URGT<{6Yf>9g z90*(B)d3n)H(`PKlXUBH-CF?=v%2p3n4+RqM6qWlH+S-n`72m_>RFSV=XTX{_8(#? zgi>7>fNfT5yI^cIQoggm$Ohi_NSC~JTw>=Gb zsR0Dt&Di~2C=eE^^5w_<5OLvlk1XU!VN32<0$`Z(p~Hervqo8mKPjMx|TmoUgni| zy54iO6l@}nvNK`@y1w$ZrnXoI*Aff7hS9p-^fGVdZaeL;<;|)~5Dw|dX?a(6_+6jWNOYGm%2u!k zG%Gq2{qQW3Y_Zf}PrI?c>W&{onoF(NuN8p+9#kz?T3e*>Ub?Vil-6AYE?hzPx8h<@ zbSXWGer!U9iMaUd!btXpPNhe^5e^)1GD}36UL?(*OM8%ghM?%e0|2#7u)He)%_Ep6 zPs=j(zm4(2jcSMuyX1AR^G^e9NoPKgTKKpkFo6tnMWGLkDAHpgRZ-CW(6zAesweDD`&DR4%~mb>2xiLOh$EX;F{o=DC$;Jnp|aH zPAJN)?}^MR2lSk0`{BOHTm!iqot>Q)fwE@iUeLPbEg3}jd3GlJNj>Lr!9YLw?b`~>|1QD7bAdS!I zlq{5;?}sP_RI1c!dibz~+UN8vW8&Z9^?Rgae$u3SWwAK1ucNv!P9=+{wNyLnPIb|Z z-Q26(M$*uNx5znETIhm-t<+Ibc9mPrF=tNu!}d!#a*#PQp;PEgC3MHoe2zy%CILwV z>)rmDM4_9wlTVZMXrGv#oswQDF&Hr!jdESL0vG>WP$d)foRn%-*e*yOr~Wik5%8Fa zyp>A1G=-NO7ycZpp#WpaeWkat^&C@=SqeG4z4eva?gv0Pv*{S|V+9HnIhTRLI^5Ui z5-IkMR_Hm9SjltVyH2z^-@2$87XNYTv-|J6#*F`rq5saL6u>L<(@S!(XUddSyH)3= z_OJhFT%B7uQvt1GabzB>FOk38YUKIq)Gp9>{)w4x^u~skZ;qLJdg`1~0~5wV^XH?O zVwlwFm0RN>?R5*38~9FpFE=*d6x&PDgpu+*Ln4sfY{hCg@OtkOy+sj#nNu*XeP?&n@AzU$s?~sO3}+A~xTvJp|TjWDK6Ve;O&(v*Fw55p{6MEO_`S~=NlH0 znmLAUUS@X$wLc&dJ0m$;juIRe^jS4qeP@8UVyS~d7qqJHIcRO5%ug(u{ zkDd?+`h#~8hR~KPH!e;tvB+ik)o`dSO)V5}_N4VndCqsYQzC8mYNdnHtecL6xZ#S{ zK7&Z_oN)DmdhQVH>9ek2F;)ySXG(GD26bbX`#mcT;&5e1mJ@#;XBAKCK*e%z6*9x! z^{jTU1tF*Vq^Sz4xJ*^zwaKU&d$t{S%lON^6-bPb?PZoGbEMc|FE2800H#e`>W#`{ z+qk0hc`xlfe-ir^NlVcv<8jqUWGGqZ+Vmo$x8s3frmBI**WnTa@5>KT6f0eLdHCyr zCBK! zxafi3VBB@bY_iH8L(|~9+YfH;xXXdw(NVVX@G;-sq;ww&Qk?<`J+84 zMC(RFxnF2q4Tr8_>B6J9<`z;W(X#Z+`_-^6+@s6GMhG4K(BH-}UZ`WUT6OsdP$0*g z-=vs5S%}+!i(ZA_s;Y58HCYsT+%4Sj1m3WNfyR`R2fMw#9n(FTf#BRWq=HnTnvIy* zdczx}zy3g6GTMD+4m<6I1fsnpeNIQU1jyWc2Of#+e%58-<&OC_6223T=$ShuG&9iX zB!Tdfm?YF6nq!F#lZcsJLUTC7KFo^>HAXf1cHWI{bn3IsT5c<$l!)&o*b6(qFgj|m zwpY=!3h7&AF7zPtK3h(TMjM`YcQ7s^XOF@N=HE7c4B_;grnXuy@@4DsF#JHT2OS>Y z7%LX9Go8m`ynJLOoueDk*=|ir!oImh@K3t4q}xZ)+g_V`m$exJY`a&O6$o-V-v?~c zo8o#e;0!}Bgb9^?vcSNEdg*C&@Q8b=|&PG~^HrD35hMpD*9~45nOY9M{KnkgE zC+^X?^FZH((xnJvMhg7?a=8tqIJ$-4r~~cu3ls0-6@GxOBD(wpzHGN?$aCrENM;H? z77t7Ev~630jeU!6|Jej~4B?U41_<_ZN7;f@o4gfU4f8|U`4TZn85JJ2tySMS!54Mi zSo+HiEZ}x~DOksNWo~pZWoaz-rdaD#;F*#w!+5B*mYH+zeecUM!|bc-_pdIg{@g1Y zUtL|hw_lq)0AZ}(S?!5a)yIL48~S=*iqB(`p~G^}Sr^{BcXK#qKui&VPO3UN&~XGb zoQdyI8_pxTYx8|3(vB@`LxD0lU^;0y81Mfdz@PKi-k-c?!1_@3>SF#%aTm5O zsu;6gj=FvOb~%7tW!a5ANf@Q8UfvFb6?YI8e0urvvMDpg$M1i5XIs(eC1g)r28P9dlE#s^zQU zwJL0^S^gbL_oKQqzh2WThArhe^KRR_Qr0LP10d!~o%m%j)8$3hQ9mwC=HI zN?8$E*Td1LG@~ie{plo@8{Kaey41bh&#Q4DhtJh-s&D>_a9=Xo=#ZOU=%$Jb@*PAxUe5Qfo-soB4(s-6Pi&!%YXCE#1be zDSgv*A2(9+O++J$CJz{*m7Kk?9tPy`Q!nPXk4b0d#T?)vHtdfqA-c_abjHsJub zdfjb_qeZ1>hZm5P9ty>6&3>Y#%^ApP`|_AW7z-xM(^$w#2oVH4VTZeGCBKflWa8L1 zl$5&#^^QbdBkgyXNRRp$s{+>NGkLD^_DPEw?zrt_STd%Cl7N|Wxs4CTi9adLa)<2G z$P?qLNVe{OVKi587RO&opqzzx)vmEPLhzRjIC?I=OjmJ<6&=wtuitwn{aU-|wtZ*2 zZA{lnQQ4QrtBGIN13$N7G21aN^?k=67aK?Hnb;2L=Fv*0KtNDZ;xfn}TQL-3Ud)84 z&*&9gD`MiY+VM0pwPC5d@;fB#9Hy2X-uwO2iH==BCcY-@MKaTuQZX68N@x``dKjg1 z*M&M}p<8Grq4$Az*R#0tf`usf=yLW7^|EJ1DhyEa2_3FIOL=-#H62sMlwh!l35@N( zJqmjMW9&BB)*8jfTjL`D=@WU-!!=N65Oj2W3?hWXxY~~^y8(PUCiFA=6zph7#sb!F zCjqd#l*0^NcJ%<4)mg_*T6x{`b>B-w(Ro{&xTC2YS$X*B{G- zu~N9lKJWDwp>oShdK`I#du29x)tg#eEg}wEXAwU}fXs%h>`@tHoff`sroDfv9>+wJ z?OI%lomx!&?9st@6|C9V?1u~|_7X=MF}%5=Nk@V86KEj{BfEBV*=aS9k)pVZu)le5`cc;^A1``ZcW!He)+2*JMJIf_jRt zX6)qKN?X>@Fh}An^iG;Mh=a~}@`9^lBh21nJ-S_H&*zMzB0QngnEK$n{qXT&c6dJ` zRd@~t=idD62D6~wnUC2iv#*7vE=79{ggbdJS3hjn&O|YFzsz<&+?wrpUXb0Mo2XayjQ^~HIHx-Qz6c2If{AvXmw>dlH6652E zPGROeTY+tEob>1DE((`xZGVe(oTL`q8CnZ-p*v8b$yAW3TAI@?Z(MGbq!nLD6ovB` z6hcDI9eeIw91{2WUzAt%rjvDAC4?T{+rK~(1u;e;Bb@_dJW9Ub_2qtj<;kv#DC0sQ zvO>}&Td`$#9}gCN#t%=pLdlVo3{?+VjRYN)mlp*-08Hf5exQE!RXeY_=$EVQTM)oy<=jJo@&44x!pXIhK zu_&&`7GlfZyOU)yT@LTp>h?YubZU6QwIW&(_aw+V3E?lDW(?%bW>y@tx?~(M|6+Y= zL^I6Y;hOG|e7V&||3XcP{=h=(xH~_3FVe8$E4@H>oj=(xP9gRaQ@{p$1#?2_iejJf z$#oe$*bY<=nivWF*wV|xW;T9OSSY*a;j^pbRa+ zTM1EQrXS4u>?h`*o4)S4h)(@o5pdw~D`h@4##@N{&giNwLt&U|)a%*XQ|{$F@IcUJ zlpIBS2#tO2POx{v?9_qT*ukCUpdbQob$?uc_Eg>po`*(q%{_!y8v`s0@ZRxI&FwFORm*xH%aenTTO0Bn*LT5K?b#g zlT{LJwW`5vwF4%Mr}^LJmR|m_HlL2!_`FoZKCzg#5xo#dDd_5NuB`1NEGmM)6*LR4 zsT}RYZYSp06+LOpVxjYj>t9vND(tJ4Nc!eo;vHl)H&*#BEJM$b?CcvcH-kWLTk{u3 zZHDYz$$qIwH~WVn##mmj-u4c2*8u71z6RxONZEQ{>ADNa1TC5@j^pwCexL)#j z;k4_G@`38>k`*+maLvD}ikHt8z{)z-zqg?rG)-#D1aBNN^1u!yE8Jq zs#=L)XOF#;gcc;<`y5mS7u$lx(&sI4rpneER{u^U&YZ;X3FnV1+u5SRIu&w z6X)HK^!0m}qx_fx7hd*Ff4O&o1cy0D(lseaYwF!Y9j>-`B}M!t1%|J&mCQ7%V%d25 z1ZZ`9<+cv1;l4*Cl5duH$fJ@Nvf3=oX@eOmi9;$)lcw8F$Z^$`T9%y1lz&>>8RCzS zf@kT-kM2rK-N=_^?Cg#*3#ckT4}}SFI5N& zM!$J+FnXLXAM}*~2j03WSX7x65jcq_*=%8Sp+oE#dg@=9i{qI3LiCPQI(OCRIAZJv ze^@Cla(kQoOxv%Nq6RGgju8R9d;IOcuF-BxP%{}UCe@hYqXP=@|6{0o$sr5~vwyE2 z4n;x0a$e{)mBKa5aj|dVx%gH=nT>r@etw7nuxnvk>i0s`$e<_R5Q%4InpwyUFolcp zG@S$cPPI!gU`a(9(oR#X+VFm;_3U)3%XXjHh_lIVPUOvyWce>#h`_(le$YH6=lr1_ zAJ>nBvI$69QFpjLb7W5twa9W}&M0TPbvla*NSi`=W8(*=TqE?`hvVJNs$GwEr7x`P zux;p;ziRr6N`pY|l*GYUtU@p^0dY7e;{6a}W6uiZu5D^2lcUN~%U-GQ#l*OrOVtme z{7!p;GcUMWNWVsiZj)>}{u4HrB!|#FbUb}QRhCo|nLME!8X#NRu`EqLJ?zbrAiLgK zv9z}7iflK3zW%HH$poJ~gMnQ^pr09YSHCXuyxKw6fvCbysh;&HUmirN78XYgJ2*~d z72@Y5ns^1(sATQn6p*G$h=0iBIs&~_TwEIPKAoH6WR;G}GIqBmf4G7%BJf~zrbxdo zxbfxjLK2scPD#9zxSY>w@pMd@J7e%W*@q8iujy%)6r~XWKy_RV@HD@67X}o$s~5ly+PoSCC`sXvW(;=z zs&NB+ZQI7;D{*x?FamUOVEyxsZ`2Q}28w=M?;;rQT(nR$emCq-ABCTl;?V*J6+j7H5|qu_bjFNti~71-mOq^ z$+p|xDOMy-DrvRIK5~2fy4GBI4}_0Y2)c#4t}z$J`ocKvU4zd^6{Ajh2urWC=E7cr zqD)bEK%h`~AS!{UJSDzSd}2}HVMCI&m#Z+DYHd=pu2}$NPUP*R+D%jb_R6J_r?gN^ za>Q^$=t9|}aUES_8v?%p6_d<}+^?+`M)Gd?k{N3>;3Q<7gN((WT_1E^IY9kimA1V$ z!UIGvj~lN&5}4fx@E53;)6&w;A2>0{X(pY3$)&M!|NZf;+(8F9g%iW~EPYyD1R$F1-+(+?lym7g-#mh`?Nw1AHjet-Y}ctX?Jkt6|k%RYydFb*=&C)rP_P+ z=tj@KxB{$P2KPJZA@mm;G8P4gDZ4Nk$Gd+?@x|`#Y}W}c?=DN1*RvIqEGJ9NMst#< zy*Tw7jy^n7^a5d=IXhQFh`aE(g^a%3U#fcDg6#Ie-nU5qidIZtaGq3VaPn*6n}AJ& zNw@PbXi3r6_frPbn^kJV!$0iSSxma2*-3>umMN#GA?yC1wr{X}ekmE6Ybk4^k*poa za?hj$Q5cOfNFs#a$&Gkhvksa!dg@+aYF_;-!SRGdRX2nXp6=`@?!TR$@nH;D;Q*vX ze96hhvKMrWd99}2G~l=BSI1({MQ(1cEmwbwMw!!sT^QxoE12rrXN`vRD?Bf>=joFt zpQE2Sisjiw`u9jmmRMELfL#uY{G(A>fDIdyfknK_qv8`6Sv3WuL5oXyLL))(cCgIC z)kV{l%=LlCNE?+f&u5-qXfw;b^NM`-^hhpFLUe=;!mO4`enXA!zY|e#{t{8(npmAD z3@|x79fJR)qR?v^dRHu|ovuq@2e*=!LKS?C2Za3Qk?j0M;WvHa7??QF~Ok@`A({6RMP{1mQ<+Y;g#!RhCe(Au~ zS@D(13Gw}YsHLe9%XpCf%SrJgz-!~3WI8)x!?n%IkWpv<7;V1Q^pU_6PpOn+3r%YL zQ+syVhK#q!&$lUtV2Z@Atdau@?>NaKmz2} zRN?;fHoK=Uv-M$@wUllumnIKXpweQH>Y8Ws;%eU5E81vwa&kmpsdCLS5Qf^S1RT~97{aO$j!LTbs7`YvX~MN zWff1jI+M6#R(k0NJaCj;+%wHlC(vvN=n1eF9eP#$+<20Q5FU{m{izjk3?2SJOh z3@MUIS3U&ICau;v6h{?tfZ^eGbvptg=$R#m_ouTAH_Vp>l7IS|ceDVvSb|P@Z&K?H z-9P{I@zKw~$TC!lLkr%i*|lY*C6Jdy%;mF00{;Q3?E+sbzCJLS^V<)29Ig$8 z`y4U7-MY?D>h8I3qu>nq60>rZ(Euiw(5=EMm9mzW>`!4gSKHI)MSU@h4|ab0R6vT)PORjGv# zgcZ@I)?4#Kc5Ke#UYtJ5jgOqQWzMSmRz7Af8_e0DPMWvXmY=(s=?rva&g9l2h;%}@ zK@D*8l2tswh%Js{D!u5q0petasxybr;?cbNKdbRUzH^||WyjU8JgYSWHk@28Q0O4K>B{?=w+Mn$VlyWrdi4iCa${sJ z9K}{F6#8}bp;V0VH6i4W#Jt;CsF|?{I*ZMsw)M~3nfD$1L%|EGq?f|l@DIG;bXt4# zb~g=JN~TBBkLB4tL|y{bWYOMlya0$dl@~{rsQo}ewKl1bww$W%im`V8qN*fmZ_uqx zmB3p<>N9ys)0>II4QGV#n0sJxzZ_)+3p+^_2mGS;i4sOks6v*75X{L?6Z_pG08*Md ztiQge{j3?`AcEFV&f%mh1KJu#_Fx4XcGPbz`T&UY`JuhlS?f2eRr|7E{grtnLLs9m zp7*4ow<@1uLTfV}J`_C*qg@4jZCO(6QkAAQ*!8M^Zdh0;Ayb)xqERaUeG?$u?45pY z($*tzNBF-`ckTtW23M;p=>G=8jcZSg1I=ch zBj}V(t)*ye{Zpe$0{gA%)ny@E1{Q(S^z@vKpH1`b5W;V60e#C6`279XZ0SI{QxoZg z!^E48Ffvj4$ydqDZKtiN0OR9XBY|7Nv7pVw%PYVXR|YthXw@VfvWMTKLwsUgtpypa zgkbW-|L+>%{^SKmClSSulmCM#rB!N*Cw{sTb$WFLkTEi4pQhgfOSx;m*hn^Euw|`TNu{w@R*OmlR{rf3O%9{=v=DG*!8-lHX1c% zjr~9q63^*`G(fgW1GEK?c`#YnIe4ln)u|h#v_tjyg_xM0T1@gI z9Jnl!Q|7yWTF zr+uhv8e`P|9BCn8kHMxn>GiU@ViJ7{0mZqru;qA)_4Rc8!eFdLC>iG=o4DhnpLd&24d z7+#YCzDsZo;JARHQBw8^-B77b67y01{b6NKd$*>LVZIM5AScGeaU+e1cCY4 zHC?z~M}=h9#{+Z`Z|F*dPVJ@)oyQjg*uIqKGWEPxXQH2H%dHrcHXW7dceOU*5fjkn z2FK-L37}yKm^R|>-w}^HtM@MVu;kqd-QnY0VceuNd7*bFV7}Voq+J0yf=<=>_$H`?=n+=mNSKaN=b6RAbF9PS?$PU;XKEbw?P$e z8Su3ch;8AoQ^=k?5)P7W2i2$RwUz?vQ92iX02!MK zV9G>Y)#VT=V_1MB8E#Ptj)>QaX0z#)(Q! z&l`Yb_J}tu|M9vGzy@{R9ks0VDY48hSa7yr_Vly>MfRGUzR?XZ42J-!PqdfdVC_-=F~|?XU+5**jEFy}?oVeaQ-bGi7LuCPnVyKIE+;<4)YIug)Q^*sru>cKnzu(Q{- zuSuKVbIg#U>k`me*;A4djbcw!-_qs?7*MO%AcXhV#gK=!qqGgA_Ht`sCtS&(GKJ@b z1Wh|nf~id+?z_Z3mavJgwlDV4)^dCoyDX-Hw=`orCRa}?(p=GQov!fw%}~6kL|8uZ zXe!!oV5_)kE?4p9-HnY6mNV^+3~}#RrPq6)H~+7D3Fb4jsCr4i_0r{kV{_&3S6}8l z*xA`R0x}HCHrB89>95l2Us{K^rwX5QT3ZO!a=i)(XW`@u#tZxV?N{DNy;Mr8YfqAUGqJrAY4C$RAvIgTYi2TfRz2 zjH3nVead4$NYpL*{puyh+vp_`(bk}44aX-g(w#PsO~P*j;v4tCr#M5^Rd>j6NWTHKLY3MOzX8jyIbY5#oRV1{4UL=W5W@sC<=vR+FNX|Cw7oW-5;Myq^s}X~ zeNJ4g1L}AVN2Y&9MpS1U)n$>Xzyv|VeUl&ee`pDB-@ux*ZH=Lb7WXC-|D#;r{`jYn zVj~HW8>F?Tjh~>m$7*(&c2mc}QbQ0TsreCO2s(j@)@K(1UWd+^e zxAlTBstf^eNm+_|Goq@n6qKPqa=;X{NsUbhp$&FMM*zlMFMR)52o`I+P>_1=<$4+_U5;IAPRXr*Z;5&&;V+URVbCDU$#ywpr_@=31XkKV*U3Y zDz8IQ*Jn^vK!aIHG8@(>qK;+9k86)cCFjkC{p8sB&YO)tZpo;@1Spltig~9)zpCPb zW-C0rm)@7c;6)Lw7JAhNFO26=2^aP{LU2Bw+O7L{ap7WL-3hOI_(-d`HS;sh8RcdR zKqO{eRP*(kG2MR4Whh;$+^F^I`TDrN8?C-WGj6#_Cn11J-IskDJpbLSUPm6S02JkG zmEwQVRvZ9wrkc*cab|ol^;M_Hd!oP4#02i)-M_fv%KEQQxBq5b&H>g_f1#+=cB{Fj z{crYzgM;rH^6x$sigQC_f5%|k-J!neD?|%J$mp~Y^*41ex=6C9HWJAy@S)%tbVc_l zc7hq%0Q?c#^+Lp5rN&ss&dYq(mKzz-C_IQ6XN+UBORtl>TS{HZ`RjY+x{3;>Tib^u zxS}b@sB|U+HQ=3L1LSn0YI3u|h1Tu-EA)7z78U*kJci)oZV2fArzMT&xoL{VI7s5)56?B!-VYtn*4`nuX_UQ0=qWRo=` zlvt1eIakr%R;JpPEF2*KXf%MflDG^1tei~J(WE>EK$JKO`cJrA?jOw9t=6)!3c(T` zt?XiVOOyBc*ZV06|0Y!eilB%7uNgd%Zw=)k;M2^aULfV>?XJgwm*xBSAHm1D5a9~z z_r!5eFi(YG|5n&M92CiF=dvvmMHc9dojj0b{ZC(`y5Dq#Z7TP<@LZiM*pmSgFpdz= zb~4SuF-=27`tyamTN*O&d?yya;gT)ijg-fqL++Eh9QB^#2?JV2{XMN;CS#OJ8etN3 zahFc(4M%ZtvoDuty#m$5@wQ&p1wF4t51V_&SE97!x7G?X_*1NDrmJWplAn!B0%)5- z)AS44vI2#$GF8pW2Xz)6v?QBBw<@#%H@=2_(fHw%PM*3vGLZ*5!Uo~r8Vj6*2#~mN zb{3xHJZqUfP$e0`vN5^J&GB0edt-XKrN4N{pW%P)uKL8%OQKqy^oo=}82D}i{Tv+) zatjwms(pwToBiJhG=>Mc@H2T*Ourm=eNAxt1DJwCI@yIK(c9q#>pAI^fI zp)V=}@MIcFuEqRLG9rKxw`_S$C)rjrRjKPW&`rRY2K8MhhnRk{{EutWD-vRvjsIE@q(4 z{AW+H`vvDxV_*MmGErEpEV9v;>nf?Rvpx%au)&KiI+IzDB!@zC^ zAeoG5JgF!i+A=`b){c_=cH|lq9hcz}qwPYt7{Vlfnu?%*;H0xKySpD4P4QI+@c?;t z8KFu~6|J&h@Tvh~1O&LO2(o2@YQ1djRY|+G$&B5p*|V?{ff}85ugdj6?KmHNf@r`5 zCc0Kr=3s%LzVbED>}R{#OUb)6ugUKj8s;vk#7)EGZBtlYXne=lx@956UCA!X(b>Pw z%O{X1o{gY~3=zx3fxpVnJe0b=vp^lKzZtYhBYPr-^~K_l{zl-C3oS)!`!PWI_;d zCub{6W%UX+Kjh~dy5(lhoUs2SwCj>OAv?Xlq3>M3+NAIbm;+cr(>=|CaZF5q9}xpx zIE5{>AP#)E{lBtCuOE&{C^4K3E5%Fz9#Pzj=Ke|ecgeWzcY6EplH;Hh@J$z44&}Yy zp30i9S)2{zmaBpYbg+8;egjT~a1jP+l6&Auvn@bY$I7V0_E%O%>xkre)YUFr<%%1~ zD3Wj0D8*d;)d9%b*qq?)tjYY{-WCPSN#xm3v^Lb4qs?Ma^F!jW_-{hEUwqz|1=Uf; zd~E3wpuWQGI9BrJ5o$$Bg{=1M&ZJe0?Y*LS)*V<81;Q=~0jeLnex8XGmdEHZjw~-? z0BPe?O<~$WK*#5j$Y=%}Itqa4lyNQ|a4Zl(w`_%?c&ui(7WXLG7Wahta%em2-`pbI z9^B*;2j(gD-i$ z>B(ro*c4|B>GZ42*xcN=&Bn{?Q}E2|I&CLvFFt$lm1uC?bM6g%?%2u`x~Z&dyb`Qh z8v@Gn=Nr#%@51$il+aT+y}?<3b|J zap;oakWnfhSj&#aD{U?_LFE+y9AnSLijOCCt-tedF#kN(yQnHCV-_6Nb@66Wq*_n1i~`$5J4nSm zgX0HLFWB8bnArkWkNiUw@{$i>cSOZG4zJ&E-(flCAwUdH;R&cbgJTy5t}XXPcFISR zr#OConnq+9Ibfg3MMuYZYQh|+ zU1BVyxOHdcbGs9}!aLPkYP=3fhOhV2bL1yQN-NTjt3qvZs|h*d*tpcnmQ2m0BrR}< zg1D?0W1tO`)FWTSXD{Yr9WXH9{;74q`X7;9bt3)Eh(;vQ2KWKeR=*pPQ1EIv_`D?m znNCXqS}j-*d|0C+N3!cwci<;j2Yf;$26zDjEl`vOE?iHe(E86}RA%BdbmyHp4Qjq8 z{oO?g(GKUVtA}H9PoiuDaZjA5r5J;ME5h-DQ8gVvFd2uuy3OSQkI{`U?@GU`>50GxD{2{TJZnz^_Fo_u6_46 zDhQ&2v>>6NbVx`HFd)($(kY05Gz>j}AV`CBGZMo{OE*X>NOzY=w{$(n?6~g#zV~%M zuiWex`uUuk=Mn3-)_1uXAg{0|%6n0$pCXlUzE~G)l(VF;%XykfJjAO_&u@NnrP_8@ z&S+S4j$TZ|mQztChu3ZhT0s!L8c!4USOMJsM`$boJ)&0%G>uk+FNAjo#}^dS|6K{J ze*GvY%%>OWNf&Qq`BxF6Xv;ph=|)RRO0JL;xnfPj1{HXCvb4Q6r)%SF>UJfUN{fp* zYko$_t?6hgduu&R$7R-fOq3=atGH_LFv>!_LaK7q=|ar@1#{CD9K2lYS&8K-1Ny{* z0*^%tNLl?t{7Mh*I#sdKSTcpC2x+fqC@k}HPd#%dvmUEX&*x?n>~j!cwJl*^9h z6(hOt;Qhuk!aAT=)mN!$zBI0<$}7E>)G>%e!vFarYu9wGzR=oFj1I(Y5pkFZJQ0uH zT?Lfo;Gsc)+8`MK#^9VXeCXnbLpJ!Cnm2Rs5{;Rr18$ZtImVYWc~4G&;byc0Lbr>6 z=o~XuaT)qz$D-L(Q=*>wDWL-&cen&-%X?Q?y9JR2=N2@FK%+nhneq_bf~!;K%2|JP zv5orvUFH-Zp+!5WJ@UaX<%z0Yf<2gf{a;}-2K1SUbHT~NFzsm4^XNG5(79y zg6?hE)U!mRh2qG6C1ktu49x_zc=^Pz7} z*r)n&>d>`q07u}9t`6@9IL4fU0tGiJ=doA6(~jx_&=cdOn}jdP8Xm4~ZnB+A| zpf)`Xe-3DfG0L3_nlW5dYq(ykX!KY)4=MNR(MFux=8UC6h_HIq`LFcu>?z*c5`|uE z-+R)4A*05LPMkB1d0-GFr$~mV6dJDy!VA?nt69Mm;8av6d8+LnAwgo&E-Mr-%$b(n zA8F?3R{8KljWJW9QsGLE2pjt_>}))8 zYjBZUgk~ad(|c=x*VO7q;_S4LZ*R(%lD@Dpru>=>E|sN|2gAFQ)2g%nlYN`&IEtaZj60}7)x7Nr{V#<7fYCkFa*1!gptJf(Ad(5%k_SB?#PEp zI8Fl_@*2{nB&0?;*KY}i$JEqlNh5Xe+cBHWdMO(-Cwb*<4NOsWszSUM37ZX_tJ9SV zwwXTRK2b}!{;!b7clfhCG6}=3Zw`)hY^XL2=n4rOD}XUOz!EVUE68 z$&*!H8{*%YmbJ4ih{N=IEjt>I1K-hVP>z%JpklWW+@%WQd@}u}1FRVd;re=Qu2sQV zu%i)W)&inM@~X^|kNnZ>X@}a&4k=%L?J#B67MaN&0Z`SdB~L&vq2j_c5cot=E@s>v z9!u0aPJ?&sEdd_)V5v5-oDu0QDGUNv5RDxwAgx-bo2GB&7#nJPD}-x2XM`N z<#>RTnM3;sY}eKM35r&4jPZyi!ZjvJ`^Kwvn5Of#c2@1)o+jK!TyHYdZ2V=)4^3gD z4tRST+UCZbJv-VUtb<6bd@0CLeWByoG8o48lvqC?FLhvg<2?gV*vZ%S%7$F#9a4%# z6?Q8uxEz&kiv0SE>?F#v=xAXm+4Ohq1BOm?+PHygSh(a-n^~662ujEL9Fc%Mfo&e- z9aP{HU%`f4oQ!aRdRqA7S#b70>;GV+1Tx@8UGmI3?wp0>*~8*Ht9;V?1}tFL*7Wq` zY^&B!PlyG46SS-*XyTS0!N^RKO5Sbs*xa^ z9s=(+R8HRv);)lUAi|DrOSQCQJ|6PsEz`}rnleqom|V11(S530B;Bg+u<AXHiHJ=KTzhHPi7}T=uWl3k-7$xXq?qB=K^@U+0;!JD|J^+Y>;9 zid>$(!~8P%+RRWxnIPgdBUcVtmpH047?;$4f`q6O6Sb=+;w@awcB$91G}^7ZtVN-m z%3|!_Oq1Gz>b}3!MoQk@q{OQg2!X~GFa)HtjM}CWb5>97@xwn1&gT|w`?a7H>+~c) zn*c4x-={Fqx6jya;`qFQtwx;`S1(cY8=Zo?Vey5@P2b+8)RZ;hTKMH7=w7Aipp*;D zY&1E|xqnE!ol~mw9O4_78}n6Zdq=yLcj@FMRjp*k{I{42R`Z=XU+}s$>njs)6@M-= z5R-`%A5br?Ad@ImM8RxtuljpTm-4*2YW?D6cH)TYO4=F0%g8g83Fv)sy{30Uu>7Do zR$)~kbwFR_X1D%(@$7PkCU&zT?ANH>qIRE0`^&u?6I?UA&v#_HU#U;7{8!@NA1XZy z4wNWG1h0(ppC=Lh3Y%$JV4PB~&%5@xnrww+peABNZbh7JcPiqX)!D8ZxIxrAlX{+& z+0RQ3W+(ywn_GQu;Dadf{T5#*q{PtuafMX_#(T1@Tyf$F5$#ush_qvhdefhDzfb3n zn7mJ}>ZW8W`@WU^G9DdXAF0NJTno9x^;a#2F|j`WiKp&)(Y&bE8cz}!uNE|5ETGbt zbEuxCIdoB?RiuST+)ib>>Cil{M!&oOtW7RZhAc;sT;VIPEmiN!i$0_5Tr@)_lL9|7 z>#56-Dr>iiW*A$q;VR5WtsG3(pMq)Oq8irwY}K0E#tJO^A8qn^zO*n@R!&4v62O0< zRqjgV7&ZtzoQwd@Z0xD&3I5kNw_~HjD+3ypSs}!h4$IIJ94PAZtDw=R$XY<|V(OnC zoCMpLrjuXey8bq8QDr%CS;W_hZ)7#c?&&@Ei`a1kIJqmJ{O2z+*4EqYwl`^tx`5o zKX~x$IDS8}c-~Nlizt$X7iQu|7)l&{I7TPp!9iA&!(&ac}FcpBu^e*r}zgGkhptCkl^b<}GlB9H}x*AK}~U zmz*I=zoCdQ_RBS-?b1?gjGD~KCcrhAhS^F0?(D9!R=bmnC{KawU?-qe`8JW?WlOW} zx3<{O?wx0h_buVMVbOaf}m3p~OgOtkbi1ZG^@xfn9-#@mz% zI|4w>mZZ~3H1deuM~+-`-l(%UQTSb8I-9{=)AQ-c<&&Z9P0h=_D>nh}t*lZr&ZcD_ zqxenh)xp=3L=9RB7I4kq*Cw7e$Pl>18JoRT1|TvPQ~1k48rW^m5gG8(AHa_FCMzp4 z4tlo*8#j(%y;U@S%mMzRFAN7XAfWZ@gTGgXGP4^yh$P~T!I`fWHklEP=ov4O);YS7 zlbE5p&NRK`d`#S7o8GeQ^C(05ac#a!h14>f-!_r{c*^YOPGbJhyqX6>adlBe2w7X} zWv%&fM3|Nze)BFcdpa6r*IAnCtSoCx4C(@L%y+I-`J4XTp~YVo8Bu0|8S^li-DezX z%fUJQ#7GD0qQe)0`|qOh5NDV^POl&?4I^*22Tmu;=A_5XTm7-$<)#7quv(d{E95Fp zEOjfchRpxs4a~e2G8%r=J%nb7&7vn%etU1 zHYZfFrCgnu;AV4F?O`5v#b44j_T}MmN&)hVb5tVJ^!n&F*j$NscY*%XwO+!#xx3jx zc5y!cJzy1xY;P*-<#nmERcgGAKLTOFaJ;?7ZM_pWO=7oXrNN?YUTxWmTt-bQXqf=! zs}zzUMt99tl$Ntr@t8TVGaGLEcEwprnB(({T2nBHy%aI z3)EP~Md0C;Gz6JD@W=?43e_BzuKl*eRtbfu?Id1U6+W!fxOH%RN9`jY1#mRfFikrx z_GY!1X?f6at?rDoitdbOJl6BtQ;mWXqg*&*g6&v+NE#$Bc-Ti0;t{171nN2EE>FP^GdBZNTQTsUTFIEGO0?O zCn}O_)1Dg_`ogMvIhBc&^F36df(Q@MhUF6RB#k+_z5XQ-K7&veU%kr;4mvu@j;2{v z3ngS-k~de0HwwS|%5h&;P~Z5z6jJR_Qi>Y4=0tx_ZfC(ZgC>LXYOqq`*$&(R`#e8A z?xTJyU!_!Y6+5exq#u1d&t0HSCzLl{jn|o43pAOHIN9xTDVWEh*Pr_~Z<(#1QPXA! zepwmQBn#-pGYRN5B9PKl`NW)^%_jyEAQ_yFf3f@_`7L`u?^`6ytmdp7@0+hu2JiHH z!g9sc!kX=HJfb$o=8Xn{G;+L2ctPwB9x2(Qgr^mGj|z!946CuR6U1qwxA)VX$*QLc z<4)P@a(#Q101fYVp-X*UQZ0sr&CuEDsceP9z>;QKj|t%}j!NH5*bO^&`cKaPW%y3| z$MD@11R{|C#9HoufuA_I1FER@_S<(i)5q#DP)=sxMX3>ap3V-W^LxHZQUq;dqkl}*?~pTY?&SQ?&9}3!hMj#`+gQf` z0Z8c>=&u4rwZG6Zd1o-NS1@Ll?9+KqxNZ~1naWadR|qH0f7iNbP4 z;2WSAI&GRlT7MYT88l$)n`e3}%^xJnnjuVS%yacJICdY0C^1{82~VlZlgvO}<_Kij zo}<#--#&A>7Oa;=CFJ@g%MlL+)0D~0BpiD7|M&Qhr<$Wn6-6}gvatVgy~ik{Z`;Ds z+R5Rn;b2kjL&D;viHGm33>Yy4;Z-<|T?%v3^A}GzA6s!R+f)jl_!ewx)?Bt?s?eue zmjbujq{V!fN9NKBZ6gxF53)SwlmVyey5x0RGpV?<6it#Z`aAbv9Z#x;?)jw zO?Z64MQI-l`yoLn*_U2-VfOUVs5;*>!5=y;)tm z;76NeS{W1-G#cy(#D~_gW3ZE$E&YoIaxkyC*a%E0Xg=Fo|+f8V+IdVuweAlh`{xIuVbxLcCOntR zbU%QA9=Kd^DK81#jB{!JgwOA6xiluv99C5JgX%|3$ZjjqD#&FDAN>B!s3g1vFpN64 z0MQjl{C`fQ-jRZ=5wNS@``cF{DmTb2b=CxbZKzu9GV=GT(JDcL3w5$6+02HKtwl3F zn$Vrse`9$TTPVVtL1ofQX@-TsSAxou*))Dc51TA>ckp@o>*Jf#RhG7CjPV(}I40~P zRSP3;p<2|n4VQmwW^8{^=1%2Oyq)h@LEP#t8F==E+2FOkLHb#iys@+Wd2XS*AYyO` zCKrJltLHt#)qUY8EzMWpcw82FIkD-?w9p>qtMr_4tKdC%A>(oAEqoSm*MH9eEN*1% z@vHBd)>o%uJp*aKzXmzJ(bhjs_qCL=t6NOW6)y*C%m&2O7ITsW;iNrkwa#z1m)Q?k zoLD74cbTlnk24O5?RJiCV`wmB7?`Sd+(&Ez zI1?Z#Ds=m~!X>k&#AJ`eP05_TCjR$jW1~;@Pc;6wd7z#CbM3&#&>_n7xJG*Jbm(Rd z6v|?|qzr;egPrk55sbOyRM6!Q9f39OR@QPP--;i074AsH4+v}MZItoakxtWi1a3HZ z%~|QLhef>2ajZ2HWN@mo{LPMbBn-UNnJee6i00jx5`92sljZ2qi&8^V0%U0m+X|onDE|R1w&g-FbY=#Xffa0B#$Y{oZ09(N- zziQ2>@0%~jw)Fqb(*6XWN*JeP9f%uL*;@R&*kHpzRDK=m{xP+Z=am$jYIX&odnBbw zwx6ID)4;0yjCO`&`^sdTg6*$y+>D<~F z`?E=+3Z4n^ECf*C&Qof{WXv452C(=-yn>sR37opzn^3j1uO5cPZ-ghl{pzGL`GM*I z{+5~4*Ob2MUIpP&xg$@~IH$umqY-}kJOaRCF%aH@DoK;#aQ+bk>JpZtbsin2`vKb! zv*}tBRw>i8TaW4ULT2A`-=4dbvcQ&Kr4s^NpsW(l?VqXEaO*eMi9l2GqgzlF2w{)1 z?crro0(kDI&gL|@3*5+tMf{rjkhi*{?W+G$@g9{1CPG^bJkjnwFg#Zot8?Q9Uro5X zhsS;wy4(vLFI}OzI>yGje(iK}_EHiI?|e$?;)H<8fbQec0nivWy{eBoJ{M<4$)xaX z!hVV&oS2YXpV|J+d;ZJs00pH9)c;GmxPejKVZ5!Upj_;Y`1P}$_D-TW*eF)Mwj1YP zJoliNxb?`XIALIn+1;gf;kxt|_PXGVgiQ#Zsh*d9$CHXv6`{0rJO1UL!}GZ-{1W{g z~B!bCX39<@qtTZTF_~HZDxM}Wka! zJrTQ8-*1=I4pqrbfeKZ(>6499!d$6px(FB^B4G2Rj_7%bgK=M&4>Q7*H*`ns7c z4EN1x&;nD%2mm|8gxiWVWW;R6Lf{meEpZ^BKTpm02qT}&cv9$yqo7bO;w@kKW!(#& zS1Wy6zwcm6p5Y-T+%4YU3$o~wsQht?108G!F>a=5g+r6!`51e$ta1+Q+cWC@sA#5X zqH+0UTxhyCShT*|xFI2X59g->jCuB zx#!$ZOAGfUk#G8MC@jcSYXVUD!ml6RDAqy3_ylf}51&*`RoQ5;Dhju`?Y5DvqPeMT z9y?9g7J$6_J2C<;ej?(HKITMGd;V6*ebOs(Rks)4h62-cRuJmuz8d!id9lHhWrR}Y zl*8>4c+Xe^-*xgP&_^Nq_n6?_{v5QJb0?=d;W*@Z77Y8Z?LH1A1?+b&AG1g$oxw0c{y+338QfVb(7u(rbM0ln3wp=~I|{hgmJ!4Jyz z-iZcWmIiIr|Gq9PkH2{w$t6{|s%KIa_7S9+vfD}F8sP7%!()1OlRNQ_go7#{qS3jq zLV@Y{x2sU_Csk8)qJGJ1-k)qu@l-C3HK%&c(ZNCWr|l7Fj?-8yqxEgn~4^{Yg3 z_v5RnMLpFE$DsV(m}iH2^!M+`a>n^kgRGe&LW|!l$h5w>CF0hlSNG4$P|@*;*1_*3 z$Y08>rO?fXY9hrRR62KJ3T%Z^*(ZaK_PAnVA058w%-s(-ZRnrW|4vC30&v$n{k%|J!x%THa2K>xJMBW7LDo30I}PV@@q zy;?M&*qoNRW#|Y0{;=NakP9MW$4j-XtG%1T56$SVpLQ&$j3fZChxHn-dZXt9fCFJM zI7hFRp+xH8aAGW$P0loZa+{uL;ip*ty(0nx(o{t?+Bg+)Cx2r|xd(&enURmVfLZ+q z`Bj^z;bcYT+-&SW-RjupR52?u>1>v4nYF>OrOKQ1Bkp*C8BW_es1rJSbc9Ws`6e}D zniN1Bh?0@a(yN8#-XFLu;um7p#PNxHN6QHtgF0+R=eZF*(|(SRYhULYYd=AHG^;xW z^n!W}u4kGh>)f|j-3o<(&PEPSAt263hYMYN2HIa*Ih&d6fcic(Ix1BzUcXEI;6Ww`n2Db>g}tim zM_g=}@jQJhI(s@&xx$S&k(VG#g4ex&KM^elPf1snt_=0wNfGgGpbcF5y~ivpEIicT z|J>tn(|ex#?|Wz^PM(A0 zaIsARq_J}0*~4B(uAc3xn=y;f$Ri)iuOn%6!TL8GXb2+k9sNd>cJx&%5T8fKs>02? z1u~|bFh`3?3hC?+IIsQB_>Mg3v=|5a*VA)U>h%x*zuUvt{ovGFh%|n{Fe}ny-bp%g&{OMP_Xk!GCepv+r6>#?@3_~s z({1Y&4fumn0W@Cx^}mszjcU6=clA?>qW%TJ2ivNX?4zp1@XXnZEti)pRR*)i?R(S+ z)`Yk0*>f(d#YbE_DLmL)L@IXiz2De;%+sD#&(mP?MtGfCsIBt|0&x|*G~RU_DW$C? zlL+`b_A7J?IbCUv&j=Ka>! z;>}#X4F!K`?T2!!yM<>zLV`axEq?EtI6{8rnB(yBjFktiYDC>hazi2>EjkjR<#6gxBFFLMP$k-E#efN8pDef`jg!31-qS-nJhqX0piI z^!5-odxAgrT+WmTVLPJEjkXPN=mCymnX_A0<4lx7zhyKy-@&@(E0yV59GgJ$Q`X(x znbnlQ5`cV3d`MtkvnMe4*=}}&qiO%{0b;VCZhPaPPl5)a1bcY&uyy0)95ZsG$>ZgWI0s!+R{A2k>=-WVXMDlY8!tN+T&UCH2~ z^R5}0Hgv1;;+5|uT=Q@~E_&o}K*bUwLw|Xmr*b+;Q;z+PA9MTohZt^*9g6km?rd1U zp?9?BQ}f|l;@C%zEsl&v^5=Xbeq>(ykEA--UbWw&qWsamoFS)AZ0n*jTX*tv@Adk@ zO1(+h=4DV##d^2lcxIId7n$L&Y|%c6QE!HVEZ1dccyD%Fuw7r^1xLI2dSwaz87q`l zw8-sODR_%1^1iuW1I>R_Q3Zgk78g&e8lN*I8=09dv)0uTwUOog~N2 zX4R1umD`+p*IGERdZfXZv8RJ?+rIc1I2f1A1)KZi@@s12X=Cll!%bJAv!`vA?1>T~ z9v^}v1A^Y%GKmbl>0-K4s9S37w$jN9*++I`lkc(*oBCmDno=s)oz7Kh${in@gBYvF1BXBeO{ROS^{|HW&Iux9Srjg*EMEJO+(-^Olas$QaT zl9e}Bj>foWXn9H^kDz6uBaa%vo!EW55AKaO#z#*k9O3f9X7`*91?U`&R7`-?&Y-b> z-f;pw9(G{9Ys&;2#7xt-&um$_90lt>b0>1b-j_D-El#Y?DoGJjt-ZnvZcWLsqwLk* zD1UtT`WIDm(#QN`2XmJsJu$}#rBy|sh4?=rq=ME1z4+xTHCF^jW}vXTj~!?6a;TSH z^Af*Vc+p%%e;hiZEMPvQ+roDayoHB&lR{TZl03@_2~Y}9s}VsP2={N) z(wBZr!`fc}SJ=T+GI+@g)8vPfZ?nJ!PtT0siT#$61p=#9C>$*Rgdq1XF#HXe!f#Jh zJEEdyd{dzUccorI8rgI!`qqZ?a^e8Sg+t{sm`Or>sO)h{MW*QvF1M7b9OAG05Qd@; zh)R529|lQv9=jyxSRn zPTZu!{>&oy?yOb9n>OHsCBfm;I`3c#)256KG}FK84xS2Yb=$>M31ns{%iff_5|bDd+kIL|9^IQZ%t`ALx1RD&>Q&o;GS3>t7NE>IXSjO~SFe29o?TGaUS3W_mz5A19Y z5Qb+@f~fws>0^|!qxIv_t@=3U6Bh!uLGMGHO{A2;aeY)SY`V^=EZb3`K6m_Opp8cG z+~7_7=T_qy>GxXJUDSkNXKQrYRqzm+9_HXM-iJP|RoDt(x z%4=DWBgI8wVk>rm;PDkn8&K)*^#X-vgE)NgiPQ(HTc}eqSBtByEZX)5oZB9 zx5$-Y5I&Dp;b2*+V+u|Q?vIit8!cl>%9;P>uFD-2`Hh~RjOPz z{r#M&VN%60TTpT!m6hN%F*lXqA%M3mWI<981W2*{qWoJi?ei+%0L;;CO_M>UNnkl} zve!aa5>ja!){WpgSuDEhNXxJbveIiTb1$Z(-O=~K}0q2##SgG7g+lY$(`;@J_-#xAIj1NEeHzJ`Bw*#5tb zfbXbuR-wg1_jdsSQwFGu|EvTAm8^kB*Ecq*E$oPp-&4LublJNZ;PW%sd;!yEqW zBHfsKLL`KDds&mEKhlLp;Q00mcsUR{=T2AI0nv>CIMnQ_*cLT(#Xl1fO`V6f>SrW; z`!+wQ4lH9|zR8iEk{wH@W@u8*+MLxsP}y1a<1c&)PQe;cgjulBThK-FU>rxaFD=$O zUm0s+A>eGiT-UO^MOId^y;k~4kokQJ3ZR-cg{d7pWJAnTAO6G>C~4q#o~*6M0>+Hz zFQqA_N%L)H373DXnmkqY0cjgRuM3*n2dP$;i}Pg>R0oZB%37k>N3|NXdoLp>=+@~H zhqp|0badxcwO89fA0PAC;U@Ry&Z_K-Kgbi-|1JbJ7XnVc-ut^Jc7Dj%Vu)Ncrh`?f3nHkK8fBSIH`wowTrgknyw-c!M%n44H~09;rMks7pPN z;{;PD!-)4_=DvKGCa1|!uOl;PfP0w&egVoltr|ZP6-XFrr`nAb`o~p?x$-vT*4nbV z!7b<2m^a*8DRfH>=a)5{oH0Bv4N4!)`)uQWk0E%~iFSfxcR&EgcQj zng=K!C*MC4FeZf9R2yT}H74Z)2rzk02a6d`VnUXpA&PcN0Epa>hKu8D`hir^XT&-9SH@xX@B^L?tL3LiB{E@7mFQ=8oRbEo=( zykL5{Xs%cbq-n9ji<|=$faT&B>~+eB=lQ!!ztytiNQx@GTGtZO>-so7q)Rg1+1*-BPyK4{uWh@(NgOc z0YdunJ$?9@0J7$A;4`13?a63CCq8+V)L*#lcOypY-$sH*%~NX?G<-c ztxFRJ>SX~f?3N-x82mg~#o|_MX^eI>{6*$%#>LP7*_^7eRe%ixm;hzOpTFEWtL7G? zQc2MJ@kCLN{m_y?U-k$Rg!=$p#H%rt?2^i_*5`fIY9R5Wy|5gF^wltXoiV+*F%e+) ztdte0bZVnD#7!UcZa-#g*C?L_KPGi_6|URe7_XRYx0PRr2rv;z7jo;tzC zDE9E)Q=xRjstd8xo!d0o0aI65sp%M0=KF&UMzR8lz@O`^mlboPJm}VEn*U=CR3&x( zZH|Q4aEBNP`M^YypxV&dENpD)^ZzoNCQ-D;aQ{BT^oAW|dBEr7h*;91|XO_25EH=>5KDBuHg(lTUSv zqCNynSU-~FnabAkID&NFO&1?!m2YW5{>Ic zgI^_ZI=_tV$EYc$Qwcirm+Kl~C{bQqT-1c8~k`0!!x%6qGNIp!*aX(mybQT$aL z^ZVQ;<{dy=t)qaq|M}=Vz@z&&@aKOX9XNvh^U(>%2TY|0SE#R^7KD$YE?$c$T7zc? zIv}Q)FO8P2-!#;9bEy>@^Jhuq0U76o9#j8f0_z9Djt(Q{&oD-7M|R!)8=`^Xk{tra z%@6{!d|CM@a@k+v{+&>eFwYF*53AL-4#bV06)LOVIH#-8E@@%tc%jb_`-9P9=Js-= z&LWwW)FP+z$06wLBw%r&y6NAbk)T==6|+Shpjxxf?{R53aTQN9YcO~Q1M!e1fv>WV zBhk-rtCuz4wWsEY?9!*YHd|f4JTNR*q%lwy?<)ePmJLI0S^CoJ)TDkPTxx?cTg}f@ zI6?5*;r<%WtZmZf)Rxxcv9^75-RiWB$`h14RnDGrdnd>h&*)B>aU;JeG@!a;EmWzb4 z?K@_xZktNgCe%g6#DP$4!kbZG&X=3MXh`bOjm6a*1_|p_Izf;6pAMR%(`ul+*;dDk z$kTGXawW8o8-$r|9eA9@HlBMM^dRzj(rEp6ae{n%ncngP-GCa?q=CwT~(APEC{tsqAm6?op-z^UOp z@=^wA%@0?!G?|i>bs-&hK6EuVeym>V_U;TXB24x`ZF5>DQ-P&f;$-;?pwU9<`H_Svb!^80)M|$1O zq~E;%-pnxsf!P>p7lHHFZS~VfpTlcy$WFYJ=HJJMi~XM!t7$2+nhL?xiP4#h34KmH zVr{nPL7pXyK!qqnc*3b#qi)HimR zXmMNH+_FXj_veNlYm*WO)1Lc*(>fSp(_2H}lFj7qyZ(AqQ{n;CL>W4Z&VBRTMj+@7 z%wGvQ2F~JAovYv5zg4lOExsr9t9I^-Xru)Ii#hJc{Mi*0;j;R``!&i^WSno)gfjQ!}Edrm!OLXMpi-*@W@JSh6Vtgv@UE0to z_queS_~Lj2`CSj!wq3zShtD>Zse{L;XSIN>GcMs8I6W(7=?o*e!;x&Y-cG+=&Efn@ zfh`On!p(!@rPk;^aSB~n?AUnjafB@VRAsl{xFk>ec9`zUqhNrbUrF42H1Whr zcZ%U5F0hB`H}=g(FTQ#lPMqp%8MZ*U?$o{V9>i|OUwzS%JxZ}OPhaVWQ90ou4ky;C zz27iKe=n^48OV!U<^le52sYzTtM8_1CfGrcJn}Nb=i{?^YC-7s40W^@HDhQ1defzk zC+EE3{$)^26J?Hs4o;kX3_IoNs-}t%M8flI^p|zYiD~2@mMNMhG%t zJ;fNKMxQk1D5wCbq)*H7;IGV=M61P3bm&WXE~V;`b;rD#bo3@j&8-xg&djT^jE}vE#jb>6-Gtw*BEG3%4#-`M&eXjM#U*t1|95+DY>cAAm=B;;JYz=W1Q zpoL)({|7O~O~D@!c$|v>DHT+LAV0O(m=Ae&XjVp|33#8D!nQdbh9I%2Tk9FntCIz* zlzSxvUKB|J@HUMfE-iaZutJIW;3d+e#2Tp}S zzyNLQ7_WfR(b>sC>MK2FCYqO5FyF4ld}LAZ^*|o0&>!F{$wY`oRg9yCLf*Z0KFq^L zhpcL#wF0?WgXB-BC4o6j?v3zM!ba>#;c$UF*-ikhJ=#krZ{`f}@OWk`UoST}E3ov- zem-fc>DFDVQbOs+d>>**dQ>tV$M7RPUI|TLH7%FTy<}q9hquEir^gM(X3k9tP7NC)=OxhO0#R8`#H%i zsDzm~!&)>krXDg7cOm4^CSu`wdo#i>Y>^=7wy#=a8?5ouVGL2{s#*=%c3#FA0}93= zP8=<^lVkSR0$p;ucNbjOLLLK303;j6L4SN_3~c|wS`uT^WrSdl{bb%j?s&$nQ;#P| zucg^!GGDC4sT8Vk8*meS^}VewYfy5Ar_;)QGB zvgNJfSgu6*8T+A4`(HIE75kSv)xe1bK=7v=HJ~MzQT-{4n-DbsnLDncwH2VT{`fxa z3rXwyE1vX;80;;#|7UM`x>)!JCsOo)0Lzl8iLIBl_;WSS_!|O$9e~<5k<~fbaJPOG zQF2%DSueQ$nAsspUIlQ%UcL$J1sr9aRT#4nIqqVR`5~df$`|p;*y>}ksM%muz1I$L6A;zEe9Om+N?D02L z^kUef8ioLk=0=9J%4c=?jn@D)qHwZCk3gi7YbX3v70)+F{$*+3#MwvlxxR&=T^eh_ zlrEt6b9F;57`fhq@7eX=f<94P(!8jiZjxv+kp00|U40Z3Ne z>vl%!%~YFU3DuL|KdA`U)#+ZXn|#x*G_RiZt*+XIi$%7}%= z4%4CS6M6$OBw@Q^(8I33^ScxTf8=*dOtI=UA+ccr5VN2KX~TLd=w09|$&2m{Z80k; ziP@?tjw-3RY)l&=&R_|*9Zlt4!nW!3Fw^&_IWDFOzZ`Koz=qXO$@anak8)USdq1a(Jwzl-LeMC z?E^=f?P}P;KwfNqA#$!4w7T%Uzt5*=*|V#>oG-2i%%ue)>+kyE1Xf35_t-&L7d`=c z1c+pk3=#fAS2O7s-fqt!NA5p#shj49g6E>wKWD?t<||h6Cs>I?q@?e`73J6g(%#Yu z-x(4oK2^5@(VnvBdHrGsxbd7jB!Bqaz&2m%j5Y0|^8DQ|tF5e`jY1V5`t(P@-t8qX z>iKUO5)J0&Jr>Rg;y3e|eS5F@vChIfRNch7#0BDe&ByYYLIMkas6Z!mb8}Fn^ ze%;F*AG|2*DP4jh$^Vb9vy7^8jk-1}2#A14BaL(@-5aI5JC*L-fV2|QAX1W>2I=k) z5RjJc?gr`pZakhb-t(UGet(Q1Kjh)r_qx|wbIz*`y1P@r-c$uDoWeI6ink*BaiMm_ z=^Wm2atlHfH!T;2$c`iJcFL-RycZ6A-2R14H?9}!Vgiz&Jtec@gL$gUmpN3th+1lBH z{kh$0@SDG{2XIk1@+BnbCb$)gVH`r@9lP;XeF6FPRvOJFJZ+9$QPARG@CfRD(bgs5 zxQ|XTKT*dj!RB%gl1*9xA;|qAsX3Cf3qzT>q~4oL7=y8gSEQeRa=VYJ?Q4v8c;mqP zx&FS_aaN(BJaBIMqBSQ3{lUERTnknV_z!+SwTH>;PJpp_!WJ^GB%P{D?=J^NgTi+A z%yUFR23zx*ikgmy^Nb^|)ogv-U4D~hgDZP4OhE74vOk&s=m$mk&*kNe#eI8lQu9=3 z$@|{icsf`O)p@63Wx$D;23p`=JIvkFEO|#fS&Uf#4e7+NVuCuH0c`I&`{%OU6N~xH^ZNU|_atMSD`%xv84`ODXlJ^!JM6D@R*U zEx(e2P!nOv)BDWH*#2@PM!(Wn6uuyu`>f}5p~3W5PYf9tLqY5kNg4TL%(h&=>v8dM z(+;B;zcMCH+QTXhwI0yjTh6G~=PSj#NWIWyZ_;I(CZL@OUBm!Lg$3Hr@#5nu4%DjG ztq(5xTZW=hdZ(1)~AR6peuo`-n?##=byYb zGWrvS=DuN4E4(>jXAzQ@SK3ek2I{mjZ?z|CI1bfHfk9}n9ai9%L(71gL@F`&+82Do zEBQE<*Tx@R3>%!mwZ(Qkn`S)XM1oj^)9e~FuO%BAfMfyQZII3#-U&7n`dV4iT<&Nj zKYs4pp5Wal!)1ipFA6~PSpq1mor+*({KcRiecVk$dSe6Dtv#y_6oqO^?4L({OKI%bLOH#>Ab9Qvz zg_2y(V+uo@EVI^X`de29J?z}L{X#75wT>*3^k-|gi8mgzXp$Fc&?zd1uY4TA1g zL%j)uGe}{2ca}s6NPi6$wFW-Iwmy$DecA?KKMidF*>~aM4Ts0(sVWfjuKA!t#6j%x z^CY`={5xV>HV2CNtA4T<)UFJdo=!(`rPP0TO>i$CfX5e(z<)0-V*9cI@#B^b(ch0Q z#Ib8B9_uOGjo7In)3iC2uGHZJUvE&&iCX;XOQhr5%rD{Hc z1Ovo`$T?4!io?C!e6G>^&p0eo@(OE%(72bMWOE?XeaqH8_>yAPHxQL#c}{s`LqH4@ zoEqb%O1b#&f$cM!5QZ-^v0egPF=ZF1<0qp8D*;&rU9Xt{5zQ=QEK=ZNN(yW_z3TD$< zRE6)Fn!xW*atxA|$ay|+Xo8-Qk{=~ZDM-r-OxXk_9q0qlD$*RR$fj;1Qf-++9rrBJ zk6#>(IL?AxWhOB15}I(phm~=J*re5^p_fA?PxTLw9L*#b_SqyP)B#(G7gu$lMhVM3 zXa}n_uCF1kZzn7?{tm4s%e4Ui$WKt#>Z!6V89Sh_9Wo^mlkA)p$j@HrNKD@DS(#qC zcEBrR|9jlx0&W;fp%DluF{^WXapnceJpT*3`PV}t`9sAlxrE+^i-?Q~uX*|_x5-MA z$8dn)BS*8tDnMVN&F=s%b=39%eq{_Ql=1O_QJ{9Ad8FdkOBhMq%#P*VKS`1G)I_bT zMmP2;u2X7HZEe@WkM3ivOMj4##i}UzEMPPV=;njc%cfXyhHiJqBwY77CmU4>=X8JZ z&YJgY{T4n%n#GzsWzT|K5m&Ao08vc?g0NoNGX`vQ5Gl)VwAUP94h*WtkDNw#tM1PT zQG4U@u;|pljf@CSGIofDb}XefLVA(mhtV5t37$P0(s>nakh`^hb@^krN%gC7L2%NS z_nUv^*^vYU{#gVdERwwd2BUiB!KQV}<;-n^GPk9fsCItoA)o)XH2P8mrz0GgVFhV4%GewZ(xQIl z3&hO`Xv4buoo2lqiujaFi+l^%@Jq+S(1&R1AW}w2GFvJe>decl`ep4lI!jL*mPT|t z+y(Nxe=1niZgSWhaQu%W;W~t;4GO?;L+?_Q`6lZS{N*GfUy5aIQJf9e_=0208;*TA z$q~zD1vG@LJKcIW_5j>Rh85-amDYmQ+hW)5{B(AqNhMYEo4NEW$N#8qcRkK5-huCh zY2(`aUl37GVB{*a?|uLHKRUq*>O?fPN>3_9uv1SHGBHySW}wS}azf9bz}M~pZ-_?` zo%t0fX##{EumZZqm*8u6Nn209R+Ak&l=v628<1!e8Q}O7f9NkqMOy9pks)XV>dgv9 z58Cn0u?c!kL=eB7mGba~jmOs~u@bd=YWP9IrBg-}Ee5i?DTe|qbHn3bN9&Krj|_~4 zs$(Vh8W|6uQedow6E0sBqw4OeD0`6Fr%*PO{B00zmV7Ouc;1&F_tpDJhPwu^_bvyA zfRHc)Y>&7rB;w1+uZIkP7cAM6{l7;<@IUv-PRyAkhJvIh{QfJ9p>gbxR^)nQhnFa_rhsQ zTjU@+%jdVQlEUDoD|w>Lgz0^lyB?+?7Y6yN6xSWbm8wHs$m*kJ8_sUeExQKBCo*e= z`CB8wla&IIj>lK0Xw4%ImHszNLJnrfTNhBrT*NRuQ=Iq4BAOlgRQ$Z6!DXZTip=6}15XP@}BfYrI^Sv2n$`i}J)m zf`8l%`-@k_^=hy5-e1seGX--*%0=W~=LJbee+t&CJ2Pvix2$JA*QMv)Phy^|S?oX0 zh>a55U5kFzEdrzFkFDcDdfkIZE6vRxKHTd#De<^;x){!eh@kWB!}+zo)vU3n)bTuD zIf|*OHxiuyP@3dB@B{YDD7xqV1J{1H*(LRvR~Dt@#VwUW@uiI>fxEXn7)d_fat&|F zq$N*tK5=lz#;C15nM|Rb`B2tpnPytnE6mr%V%IFff!dsnWj3R<5yDhZqq|Sr@y-K` zs-b7E>c|xxHIE93>!!9sp6%}K$qxK@c`LBCxk1IVTQ1A98$z7)vOL)`+`7o}dKqPN zffns+?xz@pd8SHcDKZr$qr;VfMDWROhIMY_xf@`bPW zHkuJh0F#cg>-xh|Pdt0KqD$2aaiL@()3+Dgk80D7uKJpW0K%Pu#W4gqhw0y|g5RE0 zLJE`%o`8>~d#8`X)ru5f5xn%j|9f>le|6vX`kr}jFXx=JjmMT&=hDGX?zKdr2yx%t z#!Ktk+nY;lyVZDh3%c2kvVMpZ$ZaJL4G$jznpU%$tMlKBKcKP9F=V^~4Xt_g>>V&7 zPqiSv1PCh7vyC5|-mmz3a6GjIn{%RyAoUFqV^P5n3EI`ngoov$27sQ^!?U+UmKslH z9P?K|Uy5vPtmZv~3>FNnu2;bA=&2v4AHbB7rp`uVy4Pcg$Q9lp6eam2b5OX9k5#GG zjndwtoZ^H5O3Rq|0NkmJ?1Sm-&W;`l_T`VWcITh#@*MYG4>x^#F;b5$>km-Bm;lA% zxG_DIpjEBLKuq=;%%IIa9uziNoTF-_X%!)+I{M*tza!T9GP{ed`UJh7i3u=)c@`}9T2Q$CTB#L) zS1paBtDT(|#F;pFmw}kg++xOUh4tx9{q+y5wd>{d_;B#9d_l07V>M-EaU4yO9hJpW?x9fX){>c_u5-KOM_~L_MUG)G7trqsk~e^ z*9&`K`9<_;oJY9Y|EJ{MWi`>6+UuzY7lpDKffpWSHZ0V%ffXMxI@I^RvFG2 zVFLq$qfNCYo);&Ev$r>=Lt~96>=)p~mT0=1*;@zb@BAM#kx7DXuC8~k$^5>CL!!NR zh}VELtgKk>br7s)xk0+9t?Q=VQ}Xx3`fK&4(kGlXrofcV`kmD7H6Zye(`820C5f}G z?RLToPVz8SN!eW)mSkT??bhq5=_u72gs68IkPc%YBpA* z>Ks%6CdN5uoT1Kl<~}>Sk=0qkmid_}s3TzV`2}5alI4hlyVg$?1-&A1TSg>EEB8G* zy+;U>JEsZ8+*O>XKkmgKF%kS|+WYeEQNiB=bjBa~F5N%mCT#lsKYK)#TD1LHrocU% z1_Ft$xv~0Hc2>_b)!6HkswPK6k|NiUyd03+d^&Ll5)?bR_g-fdT9_J&!ufe>S?-wO zZw{b!yGh!ULSot(1C4^usitDsp`{N@=~UdGwY@oz^3*~L(s=E7KmfLIew|iSmtWWM z=nG$D1^UmIL8=>AXP7X-Ah?~Rh$q(Ibx&7@;>FR46Zge36VCy=&QWgF(SQp{{9eF# zS$zJEme#<<8e+M>d)nKACV0VK5LS(q%4C&{g0dUU@!l&D^={g0MBEx3{P@^+F<_4?nf~``YALC`#6({pApqYOAnud9j_( zp#9VB%BsVy{hwXEPY__o_Gj=wb`wenVf`rBrx?&8KS0J7-Gf&f=siisbN$si7CbZK zJ4Mb&jQtsdc(*lzb+&=rR;uXRKRi?W^JRyL1IUf1%R6Jq(?^K}&yt)d8C{&x?%J*- zk-;awLldwwVG06@U0t^|;sazOUtnYy z#Zey3W)`W!v-p`!tAM?uE>edzR5nYCJplwQ|3aALZ>x$ReWZ@%ZhWdW7qyE=F^VDb zMZ+#jYEM|}t>kIQ`u+rbON*%7#;$B{B6YnCQ0;&OxwU=Txe`^2*+qwezk~F$7SZqH z)hHkOpe)u%Q=eJ>0_^TZjLp%$gtsgAkXdmmcdvBxzL* zJ66#1mDp!7=c84PB6pB9kISjKEOv`gt_llzIA>B>)`5``qnb!zw{Wsaa8ZUdo-&iy z;_YayeSus1kG2d99XWRKG=9!WMRoVFa*^}(p~R#8^R19fV(aHxdaPsw6kXc2tML}l z?mnI^h7~b2oY(dj#{$;@w7xW-0^$T}U<;X6JyuN(LH_;^KaaTQlr*PQfpAv<~%FL|fB znIQG(wmeX_&r2B$jUqPxvXQ&XRRdl0BmY)59-jCY-KW76xFgY_R?54RQOP{4yJH-Xp11By~x_YLu(7oOXjY{=x17w-!FL_qe zUN?6Zj|mO$l1~Z|ca`^(2W8SUD&^I6Cv@3}?fmU4A4lb!iz&lhp6ei&Aeb)m=a~^^ z2%n8r#Qgb`=m|XDz%m;unsJlR{B^Kd(y4GDxkMU*E;&axFAdkS9fuyI@xj+^6Hx%q z5c4->*;Jeip6aDJ;iC$<2vpdv;Sae$4o4v-S>&h={qC98KAQvEH~Gu0E@wAgwuEr1B%JgjUVu+Xy%toJNASjedwZV zPk~Q<4G}ovuOnjfxY)(2`6mt8ICvylTWL4o~I|{qWx!v4e zW|tnWFrZ@yyeJW?emJk;SQu_fF2hE1U~NORX04| zerGkMSDp7s29)DXwa2Z)6NuO=n4e?^44UL?i$&Fb1cV>FUJUD^%3EBa>vFZ1x2OKv zwKvv|Ve<#gE8jg83$~DU(fOQ0SCOoXB&S7*Fx@=?xizs9*fHq=Oc>=h%SG=N(4({FBx^9W`)EDqCf z<&MRWFBU&-njg2(eH4x1Q}QwtmU61a$&wfRhN(CPt_%4|JG=ex+0!J zxx)y9)#3Yx9lj1@gSm*#_KWvTktTXZ6ZsP}<+g9tCb+%VFr5mG-r|J^7WOfvE!aXD ziWFhz*IidX=^T8KP_oaAs7~500%y3%@uuYrYVgX?Y@6#{%3C|`7jJXddvS!vrF=c~ z8K}t^KX1-~72D<#l-Qt{6BYE1$(9)(({38z^dh@D{yO6eT9=FQOamKt^@34CJIz}& z=w`l>nbx~!Rzwq$K-Xo)0+HD`;MgS6k<6A!lGx)v&Xk;@8ROqf6yT5^9uQJ!=C@z? zJi+GtR!d1&Eaza}GdCz#tIVL^~*pVj2LU^#E8m`f42}qoV&Rz~6cNpiS zEAe{$^1=19Z9i$r-_d_`?1JKI+U$*HF@`}|NEX^h=~r@EwUfCqy+m0i=D+bq-Qu>7 znE3R%autc4YVp{nqL891dc~7Av~^8_I?0gh?bm(ewdf-EOhud@5__ewm{#45d*L?} zmp>j8(*C(q-=V3NO5b!olG(@K`rbhH*G>A{6Q?@1gj)Hl=13sYg@@i`ufsZMdCfEu$`=SMEPD=uSn$4H5 zcIRx8?O2>8^|(wQ%d;>ZYBy;eKBiLzp#;|KkyPawLwb6ix~$I={Yx0VJcY%us7aDj zixK2cA}~)JydYVp#<=U*9;aso`IaTlD1t}QbeH(qx~d0=k{xY5H=8=D0$wYt-?SyS z@+XX_1Oz(mlNtD`5*RIv>K0i%8}gTox`G3xGuNs0e$+BOypM06q?&=oZ*;)3W4HJVy%i$}ijDx<{1g`%0(bM^Vx8Te+~hoH7gJIhrb+!f;nlbVt{`+3ZIP z=Q@4hvSXGO#{a|{R!88rD*+>>k7oVm)J@bLB*d{%GnX@nQd+a3AzkNhIQf; zBxSHm-u&%DW&y?`yN!+Qloa}eK4TjGgwK(ES2=%?KH=aJ_3xZwCdO>-z{!+JPtgq- zj!G3}(tzWVsC)B>z0hcw<2msNJnOdaIvb>NQP~$Kv26Ttf1&xfGi`A zQ;04)Rhh2N%L^_}i~xBcItBqI&5(=xAl?rFMpu$(sysurbewK8{9qu8DXFYZn$EhZ z7ZVq!KJPiEVBuC`-q)aYzUfVPUcTX(Sfa9loZ$9~{5e*j=&14o(KYOkK<1b0WLzT~ zk93qou(zDWnqG9kd<(&LzDLM+m*;U%1$A3OR7cg-kO>AZ9PG-rnMMz{3487KAA-1C z!T1LS7{pZxCl_HqZoRh8#R&6NEdKw8^?qay$|P0vsoAfeH#4h>Y~Jng|L_~8#3bJN z>*nqvLaGvz1V@Zak{n|wF6XhfOTNS~MV^DWOe-^z*4EZ?D=iNG*@F%exnB5PgynRv z(l^{C|m1F6bS+12x`Yolg3CG`T-dF#fF^Kd%L688!G&Iv zX4$l3C5f_)cw&ThzKYhCmLZM%elM0|1*fC30Jt!rg3w0_<&;4+!sBfPV7^AW zD1o%=WgO6P84u_}d2O`BK?lC=__jS|u6>kfcFuH0?O6jv{O3Sj2Zjw&2=zhz^j@0omr@JHzfoDEF~^NRoyGk){m&r zW&~UsPWjls@N^H0H-Y4O=452fA*{Q}k;MJJ6@!rU`}F^~d!T?_vs(zm_DEyOlv}Yh zG562nf9`RH32@vzL2IHq+XNt`nIO6EIEZ2mU}{A@n_xGmUbkxJy(NlwiJ0^^4eDdr?=hL;{>vlWamSyPAypl6ciz-W}XgPm?@OCZOx7{(~>P&ZXi7ydX2~ zhrKcfOT~}C`yV2~!2kL7_RJKl`Wx6LP6&4gDTyF1KZ5Rm1>k^!9bRBX8l}zTZ5Ll5 zPl}3Y%|@^*709uoC+?(S4x|cc%S>^4S3h}qePF3Atl;dxMTpM7wbacWZzvp6+ zyuM%45yA>xI*pcG8eqzB#2t#6T6y!Kb#2)9xbc&$2@SzWUo)}I&YEVCT=hqJLP*7V z_EQCytiW=u7G-Vx#Flx`NH^NEo&cn)ic6raLCJWeP2t0b4}p9#=RT2?o@`Y2=osrU z+a%s!CGQOi=*&0{*LtWXlPiYDS*@8*3I2?gqUNn^+@AG<_5?%s2KdI+;FH%99l{pd z&r4$)dLk-$K~80KkI&tN_ylC^%Khw$rr$^>xu(&94v4UJd2K4e(hy19KP7wJxKQSi zl|eTYYmWw{q7ez!dg6S(c;^nyrZ&87C+cZ^ zwxXt&)7WbyYSx)+-ABIKSg0nZ?mYz=3uqsYtgu2&_a5wgrZu*BytRJP2mHC}M;3Sf z+)CrLVeY3H*t14=iL$6?VUTRv}tLoI^E(1h=;a&z;i-7CA#Qq zubkEyW%M@^WM_t4nav1Gpt?KQYM%}3t-+NO~pVY@1udoc}@iJE{q z)2-nx*=&8Lr|SaAgoL~4r}aG| zuG#6(+zh=b4gk>z@SNcx#K3&;hMB3px%`J5>>=3Qaf`0@C&l8r?@F9tv{46MC8^{P z<#7IU+i|A^d!NX>ksO`6udF)40TE?$E5!GgTN4q}?^14CDAc)ik%pSl+uG)hFDTLI z$u$E7`3lgo+R0*D*b6w;^@4>s*-5S`Thj1gP3uYlWZl<*HcdKy#*>w`-Gid_I!Z~* zfrKfsN%3hmH1_RmytAo1ru(k!5E(rmW5QsAI^S<=EUI-HTV`Oj8+IIybJjz%)-$Tf z@$=U_ocp<@mjsZP4*0x|vNBH{keOuGFHFQ7PwDp#|E?Nv@=44sdaeve#!Vl(KP8-? zLj1F=1^>Brd}ZP7cF)CNkberM#{p5i(rpCPpkp-HLqMu%%?7#ouV2O&SK27N(R=~k zhX4>Em-cSVTmajEB;1%LZukNL9%i2bMW^QCFl)7&TVK9Z|40-4630EritGVPkJdHH zRS++o!~T2vy&H32yyZ^B>23G9wI?{OA0K+oa3_xPn5ahE8>^h+lv6U6lU3hB2a|JG zWC-y1RJc$ozx7~w8iJ8+nmkeY7-;{LJo8yx88yoT;zJJ(4uVF}nOrq)JTAY_5#?ay zuu*-}`1eo$cW%c3n7NViG4od2{Aeb)7x!O3`j9{~f4SF&DyRw>>%+10AYI^ln&!-X zA~at2N8Y4f#h%`zArW1|JixA(Wm7pRC-Q-YckwhVRkNg0h7J@rb`Dvfu5c5c;R=tHJt;U%F5O|X;U(jhVj|^QIgn# z$18Jkp<0-n9})cf6s5j%LvTQ3EU!H5fNH=R*|@u)nRo8I55N23;$l+~H!iMf z9f_bHs!(O&?TYyVP#0x~M|Jkh|MYh6mq5$Fg<|Z8vCn`}d*=PC0JZvk07g zBaY2jLthC%lKaWrC`2i0!v^!aD{z!8G0#@!6{5ku0h?#q{7`Odwt6y=6)>(ohKC)u zGCn?qY6=Lj8Av*dVpM$8eh64W3asaGbqufEZjycg9(wV21BTzzb-&;2f(xy*)4}(* z>uKb@wp>$UQ+3nBP2vtHFuXFr$wPvaO+9iNrWTzgcaX>_tGJM+)s)?L0jqTqG8R4g)l;e@S!%`aOc4*sh5r&7Sdy-DhW zyr|LM4iRPJSXCShv*-8>*tQyP*1mdoCxUQg@^8bKsIju$^g-;Ofh;MQ{yxiCh%4Or zfopI!^i&J5+1B^tUgE@fPm*Ub_h%fTeS^GSpIHxB9Le|;!$)Z+>8$a^S^WL_htB%> zNO4}V@8p^_l{mofFhM+{J8{v88|2C1{>p8x>6=(mLp-5vmp%A2?Q0Z)6)bp&x#1z@ ziSuPD>l+(iO+AV_M9}*+1O;HcvbUR$2Zw*Gf_+Oz;nalqR6asr=7XRPZQ3U0HG#+4k^@o=H|mVrKGL(?9OIdH_d(? zZaPu2CO>wZ@PQrB^B@^7+*b3QV#~vYke}}m9`6nDtmY$lZf*}OzmLQmy`a@|&VlFM ze13L>4|^eqh6HhYjz=L22UP7Xf<3@vHMJ$ccm7hR?o@NqvB#QxvzIY+(AFIn1VaP2 z+-`QJm6X|EK~bRw$JWMlp~$xXk(C6O%PM*{yT7G0Hu!y0r|WjUJw`iSKO;8iU|Ir+D8y( zQ98`7HYLJS@2FiIgU+e9_t;KT)t?J(V4cD&^?+>pf-H;q)c9yOcCmh7eO)`qk6T`P zqc6=V?vv5%qD!4kq5nsJ#VGM=(`8bH+r&|+4P88R4#gbbPxSIfWj>c0`*BwNHzHO+ zLiSqUGWimj`Fs1y@rT07Bec!WT!C6Mu!FuJTs=GQAlDt|I7Qm)>`)~T`>npkElhIRXFw!?Fw|BEDxm=k5(Bv;NV zI_%bpi&@QS>GU5JrRJ`p#OgsHJcphTuqn4#0J$znzZ`$ol-G1;`^k+vp5=%85Vr?- z3`4iBRFsv)=eDaN8a1%l%F?Zr?V=fXZXV`_(-1t9Go-=5RI2S>duN8LP*2zQL0{5& z0uS^|FZ4iLf-d98VO#_1V28xg76UUrn&d0{i30cx2wB$Qdn;yNk+- z><^=MWRT6|F$>}HP^G0xas79leJ^S1cPUJ70DtNOxWNxu4bdC1FpQTcywMrYQka3KSSWo(0QNys1SdU22xNkr%>QCP9*~ zfzHmG-l@)VG-~&PTOPBb3jktbTr6f&U%OuPPfX7X)hR)b?5suL3n9hd^sVuyisBE{ zr>aytb1Y*IADBF2T5~u$`$B*FNm$Hg6rrrFKd(BU1#?MQhl(Zcu$3@Q+DR*who?9Q zt&R@Z8kIsLB+c-f*Y1Lbiad|=Hoh2yDQs)NTvqWfT5ua*HC>DpnSzwma;qs09-uMm zwl)C}5TjrJMH0MR7a7++>%Maoys5yKQFc;$$ra6ef(B zgNVdfW3gj#D=Xrxl!S} zg)EXO2{5}6ih_)6bC{H2Xm)x z6CKmgFC{V#U1M~0jopQ!k|&3fxUm{(9q8(QEM@sVD?OpV_pD;Zh-QkWyR}<%lZdx3+eeG&JUl$G9{EAL z`H0bh!v~g?m1XMg&ev3*oh^3$!4BAc&R;y5w&rA1FMbcKb7q#740@GybqPGYyk>i| z4P6j?;$k2U%CO~d3kwi0Qk=xrfq-!rAWFa z@%tKwH9D_zF6H|l;EqpE={vr4q_LRv5oYe(g|Bo^PEMKvTbm@Wy<6`7f9?OObyVKz z6i$8r?#r~3uCT50dpTCQz`+vaQ39f%FZHi|r^qO^9kJSz7eGQuF>K$78;uYhdsbk~ zXQ{nJb>X6Y$*y&rgv<8TxwPLU@}7Teo~@f*9e!K#R9g-V5B$~&c6ZhD2MT8fmE&^5 zcfr#)Jv4WT(x*E7-rUsnj5za&w7ze}Gco)W$O4!*zvKGx_vhjj(F;0|?Z zFGl^GY9d9ofUR_hzFt4K_ttEXgKz#AMrI#n0gKqn(jE-=>G~HXuwjF+`IE)|R|a>$ zm=GSs==eDjVvp)Hx7Iryh21+lq5i#scc?$6R!^Q5Jur-$vqPon@F3rNF!sKHR}3qd z{}i*rE9Xtv&kN*6sX^1sOW_`j(L#!MH>Q4DNYm8!;}*_Qf8b*RcUTFaAG8hk_@Tiz zpj5!V$T&}4HKi`Rh%21lGOY77IeSFu{#a`Aq6;{sh#=Y7bX$3gJPWi1B(C(*TPWKU zL4>Td4S1z-fRhPU{-SH<=FdH&5j0i{!iZluo>YSvSeXSgTp5`sndzEt2!i&*Wibcy zhZBFK88MG>rMekt>0KBM511FnOl`#{&!?xY^3pou9gK$s9>(bJ8%#zr={QMFzfv=w6$O;^f z4X574{4n6Lh$+YS;p08Z&J0%QOtzkK;xb?I-J95|QhM;AgkbkpUq}v!(%?#{hRavvma(n{QMjH~;Yu;z4m9IENULIzJ z_o-cc#`bg`dOdj{tootcl6wG!x3oK)w;@=TtK-}gy{l{shEcmFHi2lRO1P3%IqT%Q zDW_qyXJ!*8;!iR+i`9GfeUEzmg|i9C=VflE5n3W)#FIgfCdr}0+b&&9*ySvDxvfh| zAA5A{+_j~^&N~*~d-a>_Oq=%MLqyI-B*Z#bpWzx-bsTcAYyM&rbw;fNXWuhk0r9nm zx+>*8B%YBOUXUdIrsR`V2#he)hekwi1L4bwT~U6f{Gk=1+Hs=scNdO*uylm z=d}SGyb4cDdmbq}y|fh4(=W+4H*j(`$mv>RiSN!Gl)bPgA$T0KqZitu!J%s9g5C8| z>r%e!q}&5~>cTK}OW>E6Pde)9OfIbA%BvdrdQdoGvKNw6?PJyJYN9Ylk# zeFVACw3J<+Bvim(7&@&RXPDhHt#ZSCwSnC&SvXl8%}O^5u&18m_lD$kq(#miXLVd1 zo&R`g_{Qv95vvN@Ij&tt_$1LHeqv=!nF!Nr(DAB=Zw?8j8~g1c$jfunbXTf9RTA!< z6@5Ai)xQ@a=b`vmVNC2o@aQzf+BW+EhM73}VD}Wt#l?+!N_PCG4}PvW zi0lgJ=4m*?Li{;_n)2v_FyZthJadv%E4&2T$CQ>NzDtM;$A0@aKA*;YE8kDXXBfK> zJ=-+D>eiU74||ryW%X?cU&{j`_j7d7!+N*HtYcV2^X$!8mM2Nh9uD0$MFYhb)-2k_ zHP`q>@V|6BWzDPuK1VGB%FH@STi|30Kg^@cVY6NddFhrM8_Qz^HRZHJ4sz*6Dw?ESte`F#pTzPYblx}2~4Q*J195Mv!x->d(a4KZ+kV=F}LA#TGg zb!pSgTpVxwMZQgxLEcCQct;jf`qs2R^&zbZEdY|1V_KG`|Ap%wH@malP5`#Al7WI^ z#5BCt*ziKI^XBqlh{k)>agWQ~0vrVsofVdk>(`SRo%uf8xK{^N}xFZ0WCROG*&gYOe|@!pdxqk_B>xv$n62JzeScDy4K`Lb~jM?!6h?J#5U zrth_`<1Sz}1nlEu?mlM5SSYK~yr{I%6RUeAn`vmf??{6gub$49Sx06ndHSudD1$X# zR!H^(=c3^OnSpdBbdT1>V3DssobXrEE@FCaiyMDm&+#m0wS?@UZu+xlwF0_QW#K+X zieHEzydr0Y34t|8Fqqd3x94fyTxxOg6ISD%_Y;=wV108iqNdk5X47=-#GqaMWI)*5 z+&sMN#SiuOBgc7e*DffmXj&6UKzeKWX z9rrb$d3#ApD1cP-!uOQK_d8WN!tm&G74SS2)Rwl%GcUh!ze4iwtu!8UNWiuo%NYR> zPBlO*zaR#q6UWEws41)#G{xaZVPM$M;3$nY+USPK7 zBH3VMUf6f(<6e6MZnfjZZy!J|)+yN7*hncU5kRa)VaT=%*kzK(KSAZ?=B^Rj!yrog zru_b?ltj2d(NFCFjlh!FP1_W9vImIA^5DcE9u{W&{Rk^kh%-39e@+^Yga}T(JekLn z9ETuvpto$HELwl-5=!7k+6D+S@s`T2yWnhw?EaAHQjat3ew7lturt3A^h|}h7`JCoe#@t6_3Ppu2gzXw zuIb45T>pN>8pvX{zb7AIOnxju?2vCcdCledL{|@KPXH-U#uO!|DA4vJbA{N6x#`b0 zRW3WOQd9H7&1`2}mO)4`q~-kyktmLIx$-7?3ik_-s+T=2b1f1!-i zbvYCRLqR8sX!oK)CX{@!DlaL^aQ)3M4Z{U0?wPTLvaO|xilB?yFkFEj7Xd$#r6gm& zefZfI`v6TtxBkW8rMOLCD6z1hvFC+5pZZzK;}6fUWLlBj>KUlJ{+yXefv=~Yd@35^ zc>cR~fL|Vdd}B7lSfFc?{sHRFdL~kRxNBo|9J>D%g|Q{O|B`|%lUISa1w?lGqw7(N z0si`K4ata?`eVr*-C zLkfHV_&7(MuP4Bpy>=ks<9v-hz#&3#DxbtZ!!`eH@K>e+MnM@T)G3`x;k6!926=pU zVg~p4bUA)^UM$e#6Ax=8{G5E#N-KkCw-I4LQ~JD_!_lc}lzTETDw0`uMaGz2(4X|3dKH?Q3yML9JIFGKyt> z5B$69$|@)Cg*=J)THhDh5E_q1*SHa9g>&#BiNf6pJG1xnTJ_-ON%2{fLv}ej?l#3j z_&%7=Z!RRXPg^4-hV`M%4&W}Zic2~p5J6N3tL#PFmb$|Ut`p7DX$dU7HpwQHWmA#; zPXdBdKmJlov4zf;s)#k9eU3Zd!Cgmtp6xbs|Jh~*F$fn(7TV3ql9|b}+oNl^T+QmG zk)#V+hYUUgKsCRB_Ba@9Zp|P7ZDALSqwc{N3KKcQ@<{yq&6$sUY}xm&g31f!}k=}@SZJo_1hZ^kLa+lyv;Wk&XS;a^AN zhaW$mvrjFD?rZGWbCs%>zYczHc4Rkw7`#7`EU5Zsmt0b7kKElAhA1h=s}9OZQLBl~ zRIaV()38=m*pH%Sh|goRMg_(Ks}B_Vq7Mei6x=DZn_9DV7NT}@!?c7@8pc>YaFf+e zN_uTYa8fnSEMRgeF1}h=|AtzSD_Aa7AmuM-M-(RY^XKHYutV9ygb{q`m8IE9<>A?K zv&A%huG-PYHF#0{0awwSWCL|zMPKeYxwe&(l2X&h0`_%BE^B3u(Q(<+?V%md2ot`L z_)oJ!zpLp#o{GYVBir2+{7ZdD~BY8O-tYWd~HM&WeSVKng5gFqqnw&J3ITeXU~K|X-T_h?wZ%k z?8^JzXu1X3cKv*r5sdCG;Evw`{ z-CD%(n6~Zx>UsLtO;J4LWd&*L;T8R*vMJ)mzQmuKv+rXp@=O=O&)r{hGDRXc`ViSO z-QFW56+jN&*E&pML=evmzzNB2?5^!gIDZruwi!OQ@a=yU*83qX;PZZlqq9LkDH80y zPtj0a8*TdiTBTY<9>nKd-06$_{Yy`|E1mspsrv92!p6U*=9fFfVc#nrSkrT0=9 zy-WAw97YVcoLF4i=l+5L!Gt2e_3+AQ(G?b?8nFBx8S=;1o{boS63b6$E{tJ{wa)K~X6J)gBwGB%O^8u2ARZmBVp~nd*51RL zEHA&yKp*J-5xr!Ry|3&jG;3PqW2qsrR=%r(J=^|eLWIZpwf#ph?*km=|MB&eQB`(b*SDgC(%oH3htdb6K^hSd>F(|lLApz%O9|<2q)R}$LAtv+ z#J4e?c=h@I;9v~(TxVZR(1ZNcSqK$e-$(LaD(EV>LheEA?xrZat zBurvCSe4a7JS9&XGP`!nB^^Pk>7`%}$M3tsB}$71d8epZmBCpOMjJU;-{ShAkMHu{ zMnc1)NSW4=anH-3p{PFf>_!C7q-$s>;l|dUs`GFy zlWAOak1iEd_$;q*(~X{k0rFvBF$%Un$?94ATS&d9c6)R78U-I<GJ+rVg4P($N+w>5xVV@(yAA$I1BKs5B!{bdu77|9Uo_D-hiyP%j z|62$iptSez>YpFtxNDI5Vhkpxk5da+MY$IaHp8`OJ2qR`#&Epj;$w%3-*Cg6JZd)d zbFhF9XH7F)}?4 z`>JZ62Zsh3awm6lK zS)6R&wL5&+DVQ+Q=gCvcSB^wOHGWDMgSLMQHqf8Jrf^F!jp!B9_*9E!xWD(GSq!;Y z)UX+NpMA)WX5cuRvt%XhWqtZ%iMib_IK8TenjLXw6oDDTNyO~Xqq|w79DLII*SjOG zwf7J(yaqo&Kkvq7Wgcc-eAgs$Cw6{7ZlW;AWyTvrDPuteYtnN^A^}5zun5|vb@YH0HLRfg+ zoqyq8si9e{g=4sfB+<*dA3z-Hdc(#|b01H_z)kB^dJ_x+<^9DNw8AU;K|vswdbjg^ zN;@CrH<5szzlMY#IrMty;^k-G8kei@AMH+q_SRj@O!v9t<1p-Cu)#S@;tinZ=GHqY zPjav`+rfPnjR*mI55?fYnicr#06?<|&z(}4kR@`37>pg}N+RWNkIm&~Xsq^s)z--R zRv>;Lv8kas2K={dPJ_5$AD|W6{-TlM-m{|I|8!RmuVg9XOmhpTuzy=buivNzb>~6& zwV2N+(NnYWH%YIOK53uC(dW&&5aK5Vh!4i6(%ledPoZRn;urOY=)u0i7-aj)5#0N6 z1na7<-?&|WbjL5qQ58*K7W`Yp8KivD^X3BoeWjj$jK{ZhtqU$5X8`jw!yb}7h$~$D z)JC4DREA37<~ueY<(}!tnIvOoG2w$E>wZVc7*3n5f?Ou3BQ`QXCA|0_kpK}s8LVlp zW|}0gm|y^5WhOn)0vV7w-QGUhXj#R;!32l&qn|%{f|`+FmxZhI1 zXvX{wxf`|JE~d^=!0#eI8)sRyh8@fWU&1BEW2})A5N!VjU}TxQnFjU_XTx^aec~`)Rz~O%mp)NZ z`7?N><>h=OaWb5?CVY|s^yEwS@7kD}X>^rE9+006MGjI~rIupIlL~{c))GEoo+*nK zvQ1#t6t5VY#yt~54tbXH21SyZc`DoF(Tw|r3%lr98DLTz$`xy()+pGk1HaBDak0mVRg

i(Zi>B~3@PEi9CWLmI`^o`Pt~F@O?_-(TKw|!EDnc{x42?*!_5BIA2=6*i%|FSa1Gw|087(y} z9DqH&Xkd~@Dc&P@^g*DiZN2}Zg+r|lIYsq+B|I!DdZqr;(_u%8T83N1*fp?PM-(FW zk(P>!%lV7tByW@}U!5_=ZkPzH!vBB}7vh?ptzk;8a`u18%xu^tBu5+rsSSz0+wFLf zPd|C$%0EO%5?>nCm3J5vj^pCfzsI9k9A?-5GrLCd?L;iCHa<+v@~44`9^~r=X};7t zF>QCw1FPvk2;YGyxJSocIG-Axe3@6w^azt3vkvx(3u) zPgL}TZ`l6y((FmT(Ndmp!A+9R7Dv`jCT0htWQ@5lv*_r5R+80ofL+iQE{y!+qame5H;=Dl&?#%}Am_kLp5&zwaAbq#0Pj#0qv|gqGjhi68u20rF zpoW+y*g=YH3Jst}KiNA_@l$plx8~;K7ATaWR!Uw$K4nALg?L_(({;n{)i97hi3vzL zf@==UqEf4-kfNPft`|30KBfoYg-fWeqr4m zobdE_q8D-J-w{b~Perv9JtzpLmhgL`I!{f2P5R}VPj;3^66?e|dMvuKf3x3W#5_7vD_&V7Rf*63JG-Yn**N`=X59xu?~y zD$d)7+mrm#p3fxG(dQ10j|Xv^3p8r${d(7u!E2gAzm7 z-h{nsP#cs=df!gp|6Y50$oAMNTdkB{D%(90nsfwBKgZ9&bH`mevw>Q#sB`=K-iIf7 zLZPS0pX}ZPex>ZJbqfOxBB3YnF+IAjuFk*OI0);1#|8N9CZKiyvm!Fmx^CCazC&!- zkNP8o#q5%V!>34tD$x2W^GNwvuH`+s3|YN;t0u@G-$PdHqjf^2uPSS&{G~m@uoo(z zEVkHbQp$O$Q1dyyIeGD=-A2R*L!W_;4mP&RRWt|~=AAql0?t}*1NbFlE{cmu(ybCO zJs-=;%SFCQtK~6=$A>5XopAzag62;&ky>zDD5)N-I=ve)>ux)=;_*4E2~Z|u^`wlg zWrZ#ou+p25<}51(`z$+-4|k;R7(PCU*m)#LFX)w%s=QkHqYzShdBRdF<@3qz6b~#l zzn{ISUgj-?NOFOPTU^hg>{)$>(JOusFpBqB!dSzv-`B+wYHsY%9u4MEM4wp<%(F@6 zX6A-e7?O# zVwd(FSOVowZ1-VmO#q5vjgdHE+GR`u6;O6F>ijNWkt(yU-fS^nc%r!mwp+22XgMFOnWz{Umk8ZlOE)c~OZ06)~f0Tm|XxTxAeR zhxke2E((>BEx=jJEIw>1-aTopCU+3;)M3&?@Loz=+hqo_uVnR$`I5mr?|GkIm0#5p zoJubKg49rUZi|&FHj;RZEVeYnm;a;dQF}Zau618vjDs0iLODk$T>?=7OBl@(A z1O@-LZz{ulOi+-G?cc4QS@zPgm_8J}``1%IZX~0*wh0xoUSjRoba6uT>zHSQ2pqS1 za8RePAI$r>XBzs?Ijm6jHUWxU1zd@WsM+uE9)1)uP=1y=J)w_{!?fd)W(^l-Zf7;% zxW^v2l3mvzvukeN)bI7!YFgpp$dP#W-qUGc?qXCSq35NW^5Nl_SKt_94rNx-a^H(+ zAhmcqI5-G<^=Z3&V|iSz97+0w+UP0U(J@!m<8wXC=o{qIu&#GP!6>uV=@>eZa?@1- z;w(A`%<;HV(&DFoGF;+L7vdcx`;Era?{s_ZiCwMAxQJ?5A#a5){pX0c*+$Kp#NYfS zhK&kyb{tbTrxEO5dx6n^tf%xfNrh~2mOi{O6u0%Y>pCndtS<8}r9D33m`0#?i z+_HPeXq9|Uo=sOOEbIE0eMfm^aEr+Ot)aoPZf4IIsLlV!x7533)QFx7D{L_Z-$^%* z$FQr1_*zoRYSM!TUTogAYKEX|w5ET*&Fw#E*9g`zc3Y$Ee@_ql2ux?^dIui>H7J)w z&bvSt%8#y#f}@o@ZWZjclO-}JaSJ1+V*>>emfAl3v~3HnV!Zxq_V^Yl{CF=DU^X2{ zM8FH5A9fj)Zxv1ve@IK-W{1_sHwgcpnv(g?G$>t^gv<051X+n=KE>J-#^|9a;$OEf zNPB6Sv5p>hXIZ_$FfN4#2}D2_T*CY67g^qdiNemS(IM6*oMFXb+3(34nkL`q>DqKY zTIl*hO4#DYB5W(^s2d)=Gk^LV1-^r!TuwS{$0@KJvh@&859S}W+I|j>@m!;$G#0NxidiI<(Ul7!`F!HB~(~8?8Or=%QsUrG*(+5 z6a5Rsv~4~qtWExg{jWOY1?J3N-zCpS3`5$V@zoQ(GomMPT*$AV^Ogy;pm2*{Sho`t z7Y6=D3jn(Sa2vYrB6Q?oh+gmSkPx?JQfqFGzhPQm*hVw(jyX~byl5&84vuN1FAWv- z?kN8B2VO8 z{TT*Xh0;}MADQjmQ@;xUS6g%NySNxm(=Q6I;bhil*|Jb|6D#XR@F|iY zR!~(9xV*e98hwU~=l%@$vGP_!bHA4APp${Xph&ouy7l3onMC%G*oDc%(Mc2Q`M!r4 z&B(!+e}V^68+Np(*0+Kp*aU&l*DI7lg<;yq0ZZ#ayBLSUM;~2WH4hA(UM-vh zt_3DO^3AVwSlr3{8G%`LnNs&^8O-mq623>CK5oqYWqofa==qwqq+<%^b_C{p6zbdX zbTxeGV%&K}X+K#(LEAICft3Sja8C5jfZ!3p7+e0d^f1q?)mXsw(+;!=Fg9+#_r1x) z+yDFk()1t)J15ptAUU%1k$wK16&o)au3oN`KGD!%dAUJze$H~ioTEWq*!GrSn4;NpvGF+4ia9eJ{j zRGjt_v7kg!R`z|^d_jkKb>%Y-fnF)869xv&2U%=ZpTA`;1e9)pnx%c zC>lwG`{>X83<~;r5`g>#37_5Bku_{-*YI{yu}WP^nxLd(%(1^xb^y1Xodf$l30GpI zdr*tVw;Ge|ceyTJ74J&_!kJH$p(Ap{z*{o$8keN>*_S`#S@T4($&y2!tTWEY`YqqvtQRJ< zKQHl%@0CZp!xqWDjueRejB*l{lKrIVIkN5=xlIoNsPBT}D#VKJq8mDpPkso=!^ zg=WJ$i|?D7mlrjSZY~ZY|Lbmm4ZS4^!3HO^vQw3z-+TSKPQ~CH^&!8vHaisbwW7`s zjUfy#EfPw?%LC=ml&rM%01*~*B)JH= zzr#pgWEb3Q17gvD&Srf}Rvi8B&jj_`)FK!<^O$-F^|rw8Mwo3}Tdw^&&`$pNblt=|i-G7xhPG_feG-D~y6_8oIo8?a5p{7wdhBBt6(FC+ z0mC96adC0^V+u$2TQztV`BO^k-1)O87?1}+r*qKCOnn6!4HGcddXfa|J%BaCrsX4>vX%Szqa8{maVFcaFp&(@fm-J5 z<%5NV5Pt(iQS~p;C%K>M3`pc3Djltr#R+Xfbwb6d#L%QKB)?uH?oL-@7?i3aH=oXV z@~0#DWf*eI&+C~@j~D&V2Nw54?R7{uai@H_^+o(y`N^~4x7*&o1yq9RIJ28MVpUHj zhRYDBuy5E4)5yu+np;c~9@*=<<~r1>d$%wO200gBe6mxf<+H0gCQVACHc1QfLbU}< zWo}|$(rxtlmQfhQ)e6l(@Vgs-1h+BGfW*pwy&tB%AA&@p4*)Ew0@!}3vb%-FBi{pO zlslu}EbJFV))p08XV`elP#U8@{1cL2cpFtm%}t#T)%QAV2+f_cSkMPf|C9g z7Kg9BJ9XJWv-p|SLf}q%T#1GVLuf zOm!RudXAhx)HNwJ5#$EAUFd+8ER~{EOAM(@Puiau``GT>be$@)OiYF3c~yYnKHm5kr0=ml)%FB6@f%6P(|KS<&nFG-Uaj^8<>K;5M#cqZ|ja%A>%5 zb)SAGd!3i>)KCWT06|-v?kB!!QOQjKJoWR8&>v-|b_g)8_zN#!eaqzcf8^H>{OjjI z*GdP;^&T7|9_zh6(q=T?mM+DCY{s{HR8lS zET%P#NHim)C0B|v`ir6+N2f{VMRgdW4-su5#M62Wr-;j3yBvn%e#{gc0{DrH9Ug<)*Y2TO0Dr`uCBVcT}xJWpcfPL2&#iPu$6ZVj2&u4%} z%l6QKpGPj!wXjGjmE)ex38EfcAbbW+ruw&xi<6Iefw~ULx<*wUrz#Ae6?oRS-Hn)4 zp4PdXI8LaPEi2J}etVJ@y}fh#O`K=#ZH<+Mm-?;{5juEP*mwMvr>S`l1Ghuc0Bty?v zuka%yBdwM?(fU#DFZdnR)$y`Fd6HI9F{=3V$rG%ut}a?Sx~P;#s7NgQ{3N}-z1cZA z5zy9R1qFpAhod$5akE*3LhTmcuIP6<3LPPwpEw;vMDFK7dUAz95q?rH!;xa+@rK*h zIExDXERW6pOsi{KSiJ>021Z13GUI8$Viov^h+~|TRRV5hyF9df3ah9c>eLx|x3zC+ zRjZwR1)i{}{at0AVzFOTS6dMRXgWI~f+T5C1oBL!6ZHL`2{Hy~H(zRMhJjW<#&XBo ztJ0#i8f}#}{G@48f^^%vJnG=UW_cQ|aE824TZL8=a7fnRmrzy3dEQd8hLi4jM&6gk$#2m*P8v$q zm%*80(F$!d^(sRdmT5J-FtKFD)pJGpn1r~3OD3lujH=(p;?{4itSg-AKn|bj4u19} z6`rJ98X6kH4Qm)N^yzmv49@BL)Fx#=74`2AX=4#AgG&R?+`=X5U(>OGJ1_+GZ^Eaj zsflgv{_$U0dB$4uAavl8^}Tlr5r!?+qq&$h&EalXC53D3h2|T&-8>)Cl=GV9a`S!5 zb5)x&;uCNZz~YWVi#iFNh0A7bI3$)s;*oa~t?GbR+PR}^9Sr}e-iU_C+&L6&)}eVW zvd${lzn6tP8VWDpp;8r=nk4?Bz0wo)34A5>FLt4uMObeLqyGyb7f~jw;C9;fuY*PifXFs-SK-uF|3C~n6A$4EeCioWj%d>sj z$B&yq(B7T6nVP%CzI8jcnsYy$2h-;^h7A4v85dqyOq5V<(HXZ~GMGcG$*a*uBhj&UF7SA|ymR|Hsq+Z2dLd`Xn; z-d{+So7iK-NFa_PdKMgF8j!2okb8=zzRsdzZ}mQjMc3DDy24yde3xvVB6drd#d)qc zOjwYF1d-s|7&2inR1T}dEM|)$R;e)Q=mZk&BH92>b%=+LwK8cQh?JhZ%e88`U?=?Cd+{wUIQIN6WQ#ag(0lM{ekYH!q6C=Z|xHgJgCP z+;9e#TtOZxh{ahA^m56C{EQ4J@?o>rqm0nx zs|rtv$?G)FAKqN=-st6q-42lRfC3vfI+EM|bvG~|UZu*4?A9fL4Bsj%KajeWa#E|P z+PQS@rnyJmhXXr#b2T&B+tG9mY3+PRO$O10C?wu!m@@KDvhpZer4)mrI1DisiPmXp zjbdif=`AP&?!@qAb4QFrI=}x1_Qs=~b=~FT-TMMFUgyk)2n!>_yDvNqX5+GA2zipG zFV_L9E0szMBFNQ|A|a0hVuo1Q$+&xyN`)ETt?9ZJu!0sB7yt6XFgbwgd2HvYC_&wr zB8n3GhZG12o-ZOjgJ$4D#X{<`iw+!=$L*qV1==Y2Xhvzk+X;YSG&&dDkn6=r4IrE@ zPM63`4J3vs2lx^%*Pq&*ikc8||M>XW>2$^srlG>HpSW5&j#>NLrih@JuoPafoktUD z`&UIQ*>^8woZ5q$WO0RlA-gAOAl%JY+;KOr;|bgi?@wCewx{`BV-#HUyn&Jw)NDLn zJ9bHQ-KjP*Hg1}-Yz!IstgH*n*5Y_7ccXf4rzr35dfwEE$*bNP`7eO>_K$W)n8%Ol z=vtmL)%yUW-T5GrPGVK3|EdC-cS;3!G}T4V zUJL`4MettmobfEEIMWt>C>BO6Qbsno zDDt40Lc^)O&{BnDBwm>H5B@AX8Qs)Dhw=n1#tewdCz47Hl}}PPgms{#-f&ZJzZj&igm-h3A$Q1HgY)O&5fh3r!pXSi|ovzEfz-x&X-OI_K~vS zmB(2AhVtoCu!sgICwd1#Klrb36@g4CoS?lvo?YHVFm^ThivMCDOz-h=5|85o7`69B z)i7x!tG^oTZN7dKV%~A{^Q>f-2Tv1_IHs*&KFK6)OfX@?J}`WNwyc=xZ5S;p=pt4G zpQ6*0&(oYWfM~=$IpBjjLIi1fi6wYp&f+&NzJz{T*F8eyu3VmlRloN{4GmT`lZ*6s zx$Ej_D%0=U9Ku&{8!?ge^WR*`JbmMuVpZ_sqXux5;#>@7YktWjn0_BnW-oU=5~MlZ zYc$x-qXCtm#EgICnKDdA2(}SAf$&Nm!a8<@1vx&Nw)gJWMz(b7OHh{#Gm$38`b*Z` z2SE>bsznO)CFJGq;Z0uu&S3}V%;p-@Au{poIYXw3pQQHb*y6Uvi-o#ekSM`14l^2U zH982{Tc5EsKZJ0!<>=ptl@dohcp@L9GC&at<`M@|N8CM@3#+1Cr(j7MysB)x%AZb0 zD!$7~NB3}``)4z4=aCb+PT?GKcE%-2?v`gcP4r8_BDH>9CO$`{1hIch6{vzanmBu(!>OCX*Cx3}D12HyqI|N(;4(!Fn_Qile zn(uEYrj=rj1Mf?dBmCtBUqc3ykgAw<$>@|i=}03dm~d*Atj|4!<6jL z*@wlt9a_asLfnD7)U#@;s_PT);Xzw%0%Kq@-8&q} zL+R|`GwUsRiZ3`FDpc%g&G~)8e5}^Yk`KuMuqPbwnD#%U>Qf`#ZgWN$mbq)p)F%d@ z`Ng|6k_PUe{zG#bi*1;4jQv-nKmPd8FiC(2J25L{OjFD)>R-}u3~+k$jLZvM3hs)i zhETM-@|pwKchyPmmPY}s{%Yz{{d2rpg50Y~@+t3cl;J0Y6iy^)o4UfDb7Dthxbt@0 z%clqRbA;JY|AKaS>xNd-dJ$XS4GqZ}^&A?R<B0id+oc4lg!`N?o7#I0a+=~K$@S9>-1#l>O!*W_!|<14?mzTXE% zRMcH;d_DuioB{|e>*Pnb>rZDlOOvYfdt!^kUZW?(9pPZC>S$()7zW0cG(czK;i2@y zy21*2jD9l#p4XFDR2-1m08@*>o>jk~Bq!(-fi{f!gPeb&~jXm4To3YK{r&q(2UmHl(KU~1pR6I0n zusVgRG={Bha1UzQ%PGk!l{-3QbZL!N7CMkqxJ71*#0>A|hrVQu5jyMLgr0 zX*fxJbMjLi67W9K+W+nGfrpCII;2ivyVb88 z7frlFto5b$$nJTqtG3YRLQK~y{!Ql#ny|Ws)e`xw0UC!plOpY zT^(bdePnXWX0J^3P842InX@Ji8bpbF^4MP(7)7)O68j>9-yP0^v82iGgS$W|Ju~bq zOp?;%+d8c|EykN4_yf1r3kI3~k^0+Xq|KI>ek$+&&vAb1f(mp85*4W?C#3&X4(ile za5!*-ZddX!^x5q4K(|U`a^8#YHyHHB7mb*brZJ=v4-^CpY}rTJe6)e1&j%ug5u3AM z2R?GNA5^0lsByWJ`{u?CQz`(ZXj)Jt1f6K^t({7~L5`pG+W#t9ZQYjBOtw7-Oi*n8%Mb&R8dy;}hR4STg5hL1VBp}vmM6%b zia~-9Ou&I`nCxuu?ez98vmtvgPBWJ>ZPkMGZS(jHB%|*N8?PrzZ$QQ)A|&K(2fvMb z&Nle#OBD`9k8JFoJz{Rl!Fs~<@AZb&T>>+Rf35bi{GeFeU{WhDxx0ohnUG9LK$%Mu z*uM*j!?t@GGpV%d+=JsYC_IL5sXAJu_C`mzfOX&ZViU@MLUSI8c zc9f541U+P|`e!xCF*!K>99+&{^jPO_2e~&2&#Hp)mOHh(kn_6y-Waxvz4?|nHp9@Y zuVCQEiZjIJ$Y_1~d{gu0RV><%El?k?p%3f<3zC)?0o z!<`>WEoh-3qkNhQer^w_1Fcbn%==$$$-nAY<5k5famA>m=!RWb8DsjII7P2NXJBDf z1QPQA)pSP&_?JJCZMN~l2XAAxKl7d0EPwt5aK)vpk4Ka#*w`?@oTw%+7KoOJ_jaVv zdaCoZb;og!;cXfTOeCu3AoFJv)+tQ4HewGRKHLW{Na>SbLasL1k6BJ^1_DUh3%cv&m!7JaL=Dc-1{x@WkYf8BfaR4>%5NVl89 zOvdKxm~fi4O338-&;IkYbZWIgQOHrz*xN=GeY}lm-9Rnmy_#jBd>i(|BBoriGE*IP zXCiKf(LT_FNbx3DpllN@;Fh>@;QNo7W$@mPr6^MOCW@~8m5HnaX>rl(`|0XY;@&RV zeq53l{;wRRGBr`7o=r?lycS{Z^zUSJZz`}9dRK}GYAenyq!!3OYb&-N>mo?6;Gq8h zGS|=%iz1ku#xZkb0*~`1CKk7KtIR7d zq*W#!64%eCE*h5I7R@I;l|s;@QDf*;N8@Rp23CA?$e1CMrGtdjvG=9|8in|1+h=~fB+>7}LW#Bv-cTPFyKVUR_G&WW^ z<#%|jS6!_S@jG|dmn>qvJ~*K2JU0z>3ps^e1D_n!Ct8{>!c@Ms;;vp?-3k&k+Z&7rEcF&)B>@cH&rSrm+4uaY_cvB(| z$m#0YS}n@JL6UjuobYWa}jN?Jd0b5ue~>rB>c$SMwR*g?C^>!U%DqDgf!gawlY z=7lRX70)Y|{*&?hVO?W=caS;gS|{VMf=<4pLCMa&_scCI zZ#!mZyc{cu+mlPURq~#`mmar4wBo)vVfkH_ff4A~e6DGR5;XI+(ejd~;mL`bcH>;i z2R6k?riQ7xzN-Yc?Ysy>=g`BAK4t5?b6goF-g@OZuE|^T5tAT0v05WMx#RamR>2;+(Y&`sZ9X#YCoQEWp-2a08qxX5|t+bWA$i1=Hwt4 z8yn9Th)R(u?ATY-KAIC-%;}Y#Pb(!kh(7o1xZ$7X9C}jC z%j?W3<8*Sbu6&o_HSdnjnz;+*3mUDyA})^s-ZniO`DBj!M>mXDBuCtQ5Z*H~jhTIu z!bTplv}P8%lI^DYcRg~&&rpU&i*4Q$J75}7;bu*FTVK@UzLyI?lQ3tdG^pNcIx_>` zN?c8jms?Tv?s`)OZK->B!H-fd<@BOo)!faB*c>TCpKG4g%x+ekqLtt^f=36|0I%4` zkmkH;)nxaZ>NEZW4^82F>J6sODgDpbfk({GcLeV8!j`Q3nSmjlfoRZhcK_!+K9KEa zbq~XMTFCxZ7{CN5JT3{_W_&(o{3ii=paYLxM1n+qGc+I^-T=rOy{`W04e-X`(2n&8 z>X=Z}DjF5gEQP6QArWkZdVaHX#`U#te5*ZT&s@Ms(rzgBtn!@9+K{#Am;ziO;*I;8 zPZ~u>g;a6RJH>hHm)B+ZpZaIBkzZAd3pp*6(&)5AAbIEqZU$OU(7h)+Ev!D^%({lY z$m6PNKdf;o6sJ3r?-FAj4CT2$?@ls@V?!h&DH z6L+mulh=Q2`8sSP*5rtWTiOp(F>tPd1y6TLS3-RaS*7-~V~;sw7hmF54$p=>N0|#r+2KNw^+@t5CukHtr}6Z&7zts%o)cldGJJew zsd--oAGRVHEC_k1coARgRYPHEmdPoUJbv{f2Gj9WKhzyyV@l?B&Rt8L*ou?ro$b&R z7rs;uVUIv1L2;rxWSJCzYZT`PE<`?e6`$RR*;_B#EokWw zKK(?c{VL?XSYoyK&_;M5_gdNazQp(z*p?a|Yok-cah81)<_;PP&=Qv@@8>=_c2;z3 zPFS`rlu#=2i3~HsxK;1t8D##o^DR_weXL1%E+~V1ZP?^lWODQPveEIH!~ikLYw;j7 zs_QdEINTH%mbnR)2Y-wzqcdW@8m7FFr~Ar zD#njpT=B)c`@U66C4H6>gD_s>enk3M<#w=Ke}?T7E)L%2Ggk$wP?8cvfVhhKSss}C zAJXY~;qee+EU!=_gn(ljGEf5}J~`lD<<0MB-G1!TrHj&gJ;LH zO-R`w@TCcl(=Gy$v}KTHN@y8+C5sGfGEbQjtW=pA?gJc!?usitTP+sRJu`0wRIuA@ z@mx$2@Cf|>vH{7U?Mm1N#3t5FViTK&FYnb#{-_?ZImQOEo%1TwAZKX*L|l;^~aARSlg+t)OVX zwJiE+CGi3;U&%b$Yqn#2y;egXx8S$Ycee^Aw{Zs6HUkSkX5>P7Xg);h>5+1fr^}g} z=3hd*R8D$XgL`p`>rr9HPKnDFM8O*d zOS*{b5(nlfIYI(a=@K=X2R+G%e}uxdY1iyxIZ}cyFdLd_*$0S4mJi~-M&taya2_r`BqIwjU4y*?~qen8OrKm8B|AAKT?M%0bzjuB z6bUzxCdt+0;Z0_qgp=FQqIlTzj-2~h>7?aa0vnI1?QiO?`<;%a#g-`xtgbO-QX8TQ zbOz6o%TGH4hGA1)Gy}!3C$K!&u?j;!$Roo95df92W zUu;GYx$}drN`LQF!3S${DYx$7+17_u<*{;nbx^N^1YE-8VNOJj{T-yGAo9HV*jd2` zcxz9T0F3{U^A2Q-m1c1L$J=YcrE>SV)w8Oq;vF19W3l~?kXi1;YIokcLfPo#H6qw~ zr;e>Nn;4Z$GsZpoboFME`k@0+j);B>WgedkxyBWEov5tbAff3 z?2?z-j+++M_#$o|$~kikPkA7d)|gZ=A>z-O$tWg*zfz9HMi9~{!sh<--$?+wZT!uR zwQ;st!4ap#gTU(vE+T*bibc^(y}Heo+NGsTe_;x0{TugzMi!Qygq6xhLWpP0M%K7W z1zN{4v0seq%LN>o>*m9Yrb+LrvyVZ`dh4DzmpG~js^eCAFxd5#otA@D{KqO6Z3>rj zJ2x#*<;J-TBmTENH0q%!$R2M#$7`O`Bxl;W}x#uhga-ceq7(e)AE7 zv8}5`Rx~iD^l&5B{d1;+)T+#jH)9cN<#02&NZi3a(lJwR>&Oq-D<_I7>xe%-=`}LX zeyiWnV`Q4$nw zdeb&G&sK1FH^>TO+eMiCHj|J?*;iBMoSRclmyZ(nb;%Y}jXpx-c0*?sh`7zCfA#LX zmhpL_LJ)2yCHgVHMD{bcVEr@k=j`)7#X^%ewRb{uG4{WE?Y0bPALR42hNXll^UfN3 zb{u&qhIGv5Gx+Vde?nh3!w(iNeI~P~FDdqo6Hz*-C1BAvotY;`?3f&Yh4h%oidQ!` zYNmYhj>5x)O+3mCLyx#W`ExHC=uL{RYpq*x7gabhcpQy}S#{*{?2X1)9-FHBWg&{m z%R~fkJ)*UwKh3nk%;PSSgk0}N-ZW)$bgkCrI^5@t9Zjl~hxhNA;u)I`$XewOnK&P6mmio8zP- z50R`CVOT$5*gDq`^l}ECEW#Y6hKK^LhxjO(gMc}oS;*oG^Y_+(Z|-%t?>jt^jjZ{& zW34VO+tM_@td!ZClJv>&-XS*PtHF#Iu2d_HJefT~i%6Q+8@4e^l&bS~M>RoJ-X}4r#Qrfi*BxJkDf^S?Z<}xF( zBg^d6jYifOMo55B#R)@wXWJrE`%T4_EClEHtL#ZZplNZ%mu$9fyC4f;_GAHnBf$mN z0O+UCfqB8Ssx;iOI$I>{e$`;Qtk)Cz6G=)u5nAH#@f#Xa2Z8yYKEf96x)3=g*(hhR zW(M{oD-mlIUvIAx~BS2U^yUvC>SAxy#WwbaH`F=xo8=PO_%yk_PTJ=wF#pdD2-jt* z*@zuvz!drDZezh2{!P_)l7lA^C3^tO%&$~hwl~#y$FbP?CZ=L73&%H^W~tNl^twAR zHgIgaveqo6T?n1^kKHt3MHQ|csYaF1%xh|1Km2@B*t|D9-x=6{Aa&v!=)dNCI$0|} zqVP^Fw6SfseCUB}OTY3BVP&IZE&b`}Y~`Nw`s$r9+l3+w`LUONa1iHz48yq9MqBv0 zCW)Ghk3OMTMxu%Jlejm`83s#|EC(no8C*mSS(`NFsMwd&-OVAeX1pmrCyC=RW5lON zI$Nya@3y`6UgzJ7M#E^}dF-eqIC_t3-R;!RaWS_`S$iWq$D`v_wd+SIV?6pK=!x#s zaz(yQ!sSg;{VplZBe;%LqOj(&=I{m%GCr>{v?p@&aJq%&<3eiBAI?DNNlsbq?(a6E zX_5?!2Akr&{mOU!G@U61Dc{NY>Q1$E=kvA?iier`ahQf%s`U~n|GseDI>H*pi5EO> z29+6Ob`_UBZnog4WU)tg&blPM9ta>NFQ9UD52+ZhV9JWSg`sqYFuF^%lZyIVA~Otc^OD zFAb-x#)sOqPs;uuSzj3zMce)@CDOvusWj5PG}0&`Al(g;OE)6jA}ZZ2-QC^YCEc-f z?>pT0^ZeiXG~f2vW3Jhq^ZX?PkdYy*47#%$Go^S=bp9uiP}Q3|_Gbz$WIaX_cG^FAr-e|w7}Za;AvyV>>ZgOuhG{MLyP9D^PAmQE)w zxjcK)JT!FK?p;g!uffuvpj6isF9ILO0XvV+tD%}8Q{v9$=|;#}9onN_lCTs1lh4yC zG}zg<%~4+@*qv&7ZOxIqXRFlNVSTE_DOf zBiq}DsvX61&!d7{MI^aa!dKX#pR+@~b}NHT8hR_GQUgUs?>W~ap+2$e(RLm>3Qu8N zaq5cA#H~V?wQ!U5;f|cn`LM3Cq_~>qfYsv1lpUD%iVNSMuqU+g<|*PN__My4Sl}$S zGw{f4z|~?P5$lYk>du7}Lv2Z*>R?S*jxVh~MBIyNM!t?5YKtRXhC6JE^C{B%n^#l! z0&M1T_}lU%QAAW68MEwngI_=Pug1Sxl>W^5y+*nhn#EevJD!_0;jWx>|42~V8iJAl zVlxFq@@ba?QR?)tARb?@>8hB7}W%jM~q$U z5pv@EuW^tN6D;PFhRF};wzv%7ij6d5WdMb9FutU>gkxB8X!fXFs-R|W+ERnF)!b-q zzO=3`nX(Ep3qR#v672yH=)C`Ot-S-8o!8yAQ~?$!z$%ebjI$=SgGj#OLP$qe_^QAw zi&IJAqM9bM{kXP1r;0vxhvu5(U>S-^Hp5tgvMR!dO-|dFd#!DJyY?7G3l6!4S>&L^wD&aIuRla!_Ae1q2+& z#%k;&vSFDqnlUMK`&3|RGAR2m?xbfyhl6({0oVKGG6YXP^+^?l6tm63$!OaeL%+!J zNM!Bdvz&+!-k;Zu9uheJ8+pxkl9g#)3tc@1EK8tg17h-j!Ph)7@9cjVQYnVW$jAym z9ByvxSf}-GtCJ)FEpo@9p^SL;7H#@cW@}x>cqQrSd!Vm1#1_NS2)`9464~A_B5|F- zUj^L|Wf63^45WSCgdYvq2f~P4!!lJ&3x3>7e2~OumM*H>i$CN@83X!iy-Oj5UyH&IqSJd*}Igw{u|txMa8yvi-OiJLjedSA*|Ci z)J+;ZaD@Xwf2Afl!QwE(F{aC?jNVS4?G1Y~`J)Lb+%c&`^wRQdM!@E>h`h%Pq+0*G zNg&Bb&|=sWj{q>v%44ouHKkWi*#Cf|U^_EHj&KCGiZpNy$=V z03-;zga4c4m!7daWvdW=2pc;lNl$V>g35`XqlPDO?#vep?nWK$ONDd~I;P&5&Lyu` z4dfCn*t=K2{B)x`7u^c7iy#fAG4W-wgOHW!z>O`yCV9|Lmahs{x!f4 z?TC~C6a~DHfx+o}-&4B=(PCLvB-n`7OyPkqIg90`lSkXi%VHM_C+u*B$P$H(Bh0_|cNYb0Z$Wa?Up)eB8VppC_ znOMiej~2+QFNX^3ag&jE6oLL8t);N)P!Evby=d zXY3}Gv0OmO&dvx?`;2Tu8IX@cCX5mqLxPC?RvC;DlyY@)Hn0g|?>stW2pRm!=~}Y1 z6yof~qs_PP^-?Ci^XZ!Va32w>!sN$D)huev zyvh$0HH^qHq(gQT4kSau*<^)d;DY!>9u+L?IE~}-1ivB6c!yZ z{~9rI5&{m8jZLjPqj=UYD%rCB;+*(26O9bXif&Vf?1}hkxHh-#b#*iMo(_+sxUHY# z|C~D5w9*&IUGFW9{+pR`4;lU*DO081Xx%kX3Vf?{KIGo>R5wprH5hAuHP%jF zy`Td}%GzTJ=t?LyDP!y(42lwM=8Q>{Cevxq+ov?e%bXez>}-Vt2!!W@@x#e5cgxmy35#r! z11OT&G4tdH{ARpxgRtxZYhafviy@b+>zfrLJeg{~ukY+qeDW z3i+ggqtgxr!^}mN{EaV*Qy=-|JCQ1b*8apwQSps@#SkFxD2`G$^=Ko{nX_Vrb^Trj zkNkMm8(@__edOK-k(m|OFq!i*H9cm z-Cq}w#YJl3R|z@)dK$eZ{F5D`o)Zn&vi}K`573VFuSSMG%#LpS-mrVGr9<4&-ynW) z3c&L{Gg;a()*pImdwkHM^-t?|E&*6IG_CZBk0;Jf>IXXtK~&>wtmLN`KZg5LIgr0# z{e2#i&4We6ZIxMkJooJ&-J6UL!aI)$CxaV@h4aA$^D5Bmb6C`#cphU$Fe^|Z@ZEV1 zki%;nyq^-q6Hv+#7Ml{KSu88%$xQ!~p}G)aRU3DIlqM*M!J8(89jF<4W7Ml(-UoG9 zweG@4AO)4YEE*pBRrGDf4Sch>&aoypWLb-?-OQ7@Y}I?W-j@vDTc!eW-?qgSD9i|p z?1TFM_e5!CdaxEkB_svd@$ZeOUo$2aJxCLMxNh7Yji99~!k=|*KD;6|*}PD6X?}X6 zb`e7TPglpk5vNm~1!L9Cr7oEsgC;Vq&-Rv7+3rv8LF|@P6MhG{yLXgfGIp;KRv18z zr#BdM>-(|NohJ6)1i_P`yu4%`SmZE5b-{9O5NAHlw)s4y*rTUl&^<&Ka^s4tH;YTTVPl&}gx!jcq*fGFqcr@uE;C&`(H%)` zK5ZlbnmrHQ5gW)x#RaZZ^Dcai&T1M%gpP5cnN6tqW6~ia|2b`y*!2dZC0PGszKu<^ z2N`Vu=MECA@kM=qz1voWdN#;RP3-s1Wy1`=&c(Xq8DKL1MqqzpdANvBzN z#hE1|A2{C|id?dmje5e*oSX!1?H8A&JkWtrNBmSMc3?F#?{@#xyf0kDD&BilRg;(w z$nkZjdFON`qAJz@|EEG4`ol^Mwqd)^EANqutCg^wN6=TquQ3B<9W)5Vw~Q zs?@|?cM&XT3ey?&kctbYP97;!C@!F1+27~p5R9e-xh1qH7T&!~oQ#@1tI!SftamWz z&gf+`Z#?L=3l1UVf26x{vQSu^fK^Y!2O)}#OhD%I}Mp0jTlmsPZD4G*ugyKH0wbEwkv)ymQ;|LzRFWb zf3KTKCzYEp%w2Q4fLYPdV*Qh#$4b`N!NnP;v>-n`Z1D+TUe9~Vyj;azg%U58Su=z& zx7&#On|&R&R9TV2m1FK}TP;bVB0y*&jnyR>o&K8jRyGz}XiRPCwk7>lFh>3i2(pL^fa9;~v$F9g3+ z8*w2);0J&Ulg%4t=C$@yvXcNvMF$*5`>h0ZN?@~28sJ&WEA)A1<4ZhE9c8tj zM}Lg$6FePv73t_I3T+ndIig#NY7}nz=)>!OQ~Xa%B2E@x*|6~$1d-;S$*z5^6g{Bp zjnRae!UUfFS+7pLH6{$$7cv8Zo?$6jZFd%0Jzqeu0=+Al-RjD>&%_=1(P24p4TkBS>OUA=vVH>KmBKuo z$kyaXg1t{fsQ8VN{8qvaD^%|%$>&l^0tqRn8%dx`D0Z6_(I136>V_Z_^OM83S);vK zBovp7R4x9Ewgr#^Fk~*2V*wMGKHp~B)vN(7!oSVy*QcRRu?jUAWsCnPv(dfouAeHf zF+9ldhgiB8K*8$?+@58&vr1sn5#hxU1#uFWU(@}=WH1iCkzK7f@h4i;V4iCyYG5$_ z8!h?~?{8U^fGaA=IPO$T<)0EFecOASw1i`XPm`=XnKL?=%LLcP1w1HGh_KzP<+oLO z9=q4z0uM?j)41>zpz$CBprgOHrKu+|ojWI^56!Ev*a})% zZS&qPz79hK(yogRf^WrDy_VZPO`MQMSk0a&C5S!HhvhSIgvSe z>i=TcHO~2 zcqC5Atmda(d*(I%TFytkA2Ci|*w_#j_gkFjfk|ZeKGrD;HwN5#f9DE4Rz11<)ogww z1=JhaCUP7!{T`$?{QI~1*1h8vg{ri(0$-8U63M}LSDO}4HUzhREOJ+&v zDVTL%P9`61sd%ci$|1fvOD056-HKR2O-i7Xf~$r*v!EJ(xP!$S5gQkvLl`^Jf?mTBg`Yq_x+}A=y-g9aw43YTaej%WlnAhS@vYFhO(hIl1vgPo}-aZbKJa~9j_wjEkbDIr|y04cCaX% zSUgN$!1`Xz3%>%VqM<1E-m$CMSKU;CjHEd7t$Xlb&n z-H-730C8rZJPV+U|0__V{@$54&b0m~-&~;U|6L&nvo;qLf=B5k9=;1ZB9; zOwyHG5B^DPN8nilnLF{(h4tqE1yQBZjvtP>bQ~`sbE-gaa?mlqa`?VfxIR(gvL9KF zv%>*KVzHg&Zt<&hnz*0P9a+*4`uL!tXTxTFS!IC(4?lTi4cnKXx@ z-DRY;Y^8+)D^7^?G&#(>@IZY~Rx6`e;7GsfI>rL9;3HjzkjeA>wvhl9gl8;L@V+G3 zA!nyPUItgYuvqm;#Cz8%LWX3`6&>68d_$bVczU`pdotfZu7^<0*P+1)Hzi>sa>yAN zC5_e))g`KzRU1v6#y^n>OH#jSFBQJQvyFF0c;=S`>u+UspbL0&&gU$Ap-BdE-xYER zTHG?^##vs+NObSqNm1d&wjf(!87;(4+Fvuc4qCiFo(%vJPuTE54MrT74ZJ0t`&Md$ zFi8_-xcgUcbvuQ{?iPf_s*e`}i`IqQhcd0n$ts$=pzpNXhoZN0Wb-C$l{s?cNgWW3YVj1yz&QyOP>r?+Wa|&OA`!pn{Wz zfre+&pn~8zN>Qyf#V3K7e5_K|@%BW2D>eaNmpq8Zk{v5)>ucZD8`q(M1R9kF*xICs zf$fvd=wri4CZ}Omkw6tD>~Xub&Ul0@WuZ5Sio|{dZKto>rsXP%l_yx9<)Oi9SeNhz zn*__7b4hTb7MYr5k1ICiv= z-hd#`vGCor???2yZ#it%z4bT9DdWvRXdjzD`#2S4=tbZl2W$JL;a=Tl0Ct_@n$yEU zZ{B(uZ(2l}J{KMVnfe>58p4ZaPGM;iI>UBoV0(PNf>l3^`)Se3%_vK z;K3x+Whcu2GDvx@R|jYq5x9U`aofRtCh2CW^~z`u1Cnoab(#nnt(H(Qk<+86C=kE6 zT;JSKyYv(Tu@zlWw3N;n_SrfY$CP}U3ufoJmRNW(PdaiQVLx|_IzC%dp5=yAodblm zkEtT5Pv-S)8OU@>p_bFqHAGgoJ=xIl^dQ!kna()vX=c)46>YWY4TWUDh?Sa-fB$?e z^y-)0Mtyi{XK**gy1JFm^q!kkvy#HQ#;jXJ%JHA{ROsE|Q+2{+(XS`C52><67NqH~ z*1{l+rw0$aRoJY-=6sC!B>AH-20c?+Fv%WFEabUEi%Cf-dapc-(S>kXJ5jrE*ZA$ zxY8AXtR{(cC?d}(9O5Y;SA`9Ws@(qd(u?Rk_~=wt@5M=LE{}P$P)Q&hE@4dBBg})} z3F9I4{B!+xYf$falhk88oZ#(ayOY;Jx82$u(d-5w6v*s7+BlyWjk)QSi12X273RA@ zQPZtN-GlD=n36vnzJFE|OTw6bpl#_jT;!1Zt`5@aM;+wpv!?0u-G0#F-g;-!WoqDB zeQ-@%ZXqx6gd2X)U<7bgg7!TK9Ll>wc`N=CLCWqv{2Vl%VMFjF6Pm5GUozyHATsz^o`j`<_&TCj?g&@*I9 z)r{`)87qcIK~C_B_S?sFr*G?m=(3@4iwq#$0>@$=^_dSYgH9`3`x z%niFzab~mjX7nG&l+sSY-K#>K%uxtxP1| zxN3D4s_R59D%O#SgMvRGkG}kfa29(GaT_<(+y+2{KIz(X^RRx^?VfVp_1z=strlO~ z$GkL0`v3kjXL$!0Q3MfVa81}qymx5A?u{Jzk<_;4wEoF7whH_D_w1ldUTBL;#?pd~ zIED7O^;o}SV>$H0lEdkw-eL#@V41W2ICxccZXM3?mm+zug{XFto$#^o6rwa_2bT?& zq1_+ITZ{;Y+(BS)9tL#6_BU`-sw9xxkV-)wNUKUCK@gU0P{{=qn42sa$-Gk->S-T! zho8=SePWJ6GBNRncX(mf!xj(5D;68Q4wE9X!D9%t&y8@Jk3kX78j0Gxq&dUJ=H>)( z09JLkOC!=E0eUjVjSI-lMxJk%X{in^)$zSl#lr!g?=PMZjj9@xWt@mX(%=)t`xgEC z*OudpK6qa5lk6iNF7DGZi^|b`A0JBVzTB#6I1Up>r%=|p>{o-EPn`~nmO?#6S_o=e zJQ-8#PZP+!^$mMpHA?=iq3eQCo|!As4_xY@0TPe6=yg)L02sWT>)l;gE-G#|{>3h5 zCK=HEmk)QJ>CyF#)=#-1X??0mF$z%qVgMY{KgL|3iuTeX1b`f#7a4CUxv^o{M1LSy zt{li|yQ(^VUztUO(tw!2V`tbS#>dC=iQx|Zt?)t3(xM49HhVqS>peutFKsBdmJTFS zDAMdPvmA&$e_|dJpGFPY*M5y+*1%k8g~w1o@oy7>J(8bBLt)cTMD@PD#FHBz|H={h zO+aej`t%{1&?dyhkOqUZ^hy$eU>+1e3fadq`LAKlnv?P_3M`@=T*FVOri-=g$16yY z!Kh8g%Wi3w20WIEGP1|?;2!njl@nB4`9@j zw5hyzf8)@g-VEq=kabCMh^vMN4L`hFPIG1ZY?y2mve zT>~sleLv3ugvXVdv~e=(k^4TUb0@z_$<69S+vh~&lc#8|y^~_s^AwN3jm-ZG`yOx& z%Zp!%UF7ujJFtbN_|v3lFDD+_bT4f~R7plaWp0Ma}Fy zr$iTJM{ktSH<-3ci~0H@?!%+W|X0~MUN{cu<2WO27@nFd)rmY!6DlxE}4SGf4PWMFXl^AyKO4%&VTmH z4#X@%-FrW3{~dxQGZ6HJdIW~iJ}vaZL3M<`Bo?_U+OvKI?Q+i=nG`mk&%MEKLeX06 zrF-VN5I^pf=sgFH0lbOoGt-u>|B7bfBbb&*(xZ(g=c8ES7|D>+H}4dCFM|hGLq%c+ zs@jmg%-G;OL)8gfubG;f*~{_SM#{K``Pc$!`OJ0@Gp$Lo0v9zm6{{7a)D#cAsy^k( zWG$59K-F!!=7kGk`ru-80}A6fzVFaUG52p`zWVwGpX!KAN9_)=MIue)adLGM!mx|eJi-K?iBY~vMv3f{0`c5| zDL}A<>KfB?+NjdX2qSQQ9hERVV@6c``AsJD)!C2p%?&_QI`B%E5+~mt6R)ZH+J*?O zFOt5}Xcjd=N)H@6Q#(C)_p~-767X3aI4^%Jo38e1>=U9#PV`nE;@`eX5&Jzn5ySHB zd6uzd{a5TB;q~Lb86hTp@8@FQc^cA*wUp8p<5GNxQt%m1qv?7YdE$hG{cu2#>RheZw()NH!a-;aTo+ znN2j9Th7e+wieKA zn6gmf4h){`sVAT8HB55`s~uT(E*x!HSX}bQVvNdlU0@T%>&5#qbWX`#fs^PIYwDT=2C%Bjk`T6bT9lOU9?Ro* z>JPLfs07uDKHL<(tXgzB!8669Z@jNXzK?nIUB=i{v)SIFo$<1E3*Yx^m}`H<>9xn* zx}%&kg3&DSkT8P z8N;W-TTua&?$@~mY={A?%Rc*JLO9MO8>s_kuTSqQ(tM)O+19=|P$gx~Mo~-x*hh{h zOv;p!M%c|M?mN}W++w=EH|Is;)4{b5u%?MJYGWaAKU zsN=WVO_w8;LTB!MAx=4VT!$Ys8?zuZX6KD)Q5@ub?EN)_O`@YP6zFSJ0?Uk$uq!G! z2v=Sb?a}Wg@#wyx6_LaxUqip!$fW2@=QR-9MpV?g(d4VN2T2(@2*4Tqc`YI9uU>#8pM3;Vtk+j{pubAr+VT4P&(EW<2F6`sCoCi%qwB(IfreSkU*ShZ3{mR_#*pf8eEe@wW{nw+Sqg%P6Tm zp?SzBZO~P@wk0YMn?#)7TP>uZ%}B93O|10sgpe>ByDX#89kAAH1V`_up8}4beB9{Y zVVSn0#8GLp4OvLm{0`1oxb2&X>U_P-J`1fjre`A)PmGViM|%u-w2-r{CF?$?o^`Be zm6f8H-pBetqRgZb;6_f(pYQbU*VVT0lz>P(@uH>MtXZe9`QHfK;6I6*SKGPbW0q~Z z$+&wD3_KaX=6A!ugc0PDN_gS6Enc{H&aGEWZ9koBcntcs_1pB$Q4Y<8Us9t(?;|=@ z<=w72m-B5Z{tD1Vc0jW#u{lFl(GsKKL*YpCR6F*%;%~TqdOdPE$uii>J;vv*o^^)cdMWd$4vu(q#P+-@mpwMACrfl4ioNs<8;($Zxh~ z!Mo04N%Fmj`JmD)yw3DwHo+|3N0R>Lx=6RRM~46U@nnI{Z^##5`}uzdzP^74zO2{K z4Rx`OzXF-x|9*-+%{1a~_lvhbbiQl7GG?V0#9MfE!Z%mz$$HmOatLpodif&g*qikB zvP8Bgtmp%2a?J?IE!7-@tNHdM*Aj2WYlsbPlfDaM%Y!}f>6#gKFC4#MSpD57B-9Wo zcei%%?B#G2Cx+FQ8HCti0zPj8C)PV{Vdt;Py^gi3To*(BJW+h1o{v9lF@ZNxrlVCgKiV>rYGkqr4}i zsL)lCDtewr2eN{BjK?4m(OS=hUQJC`YsR9u(pZHNYV<_J(4g~?7T43WtGID9yUxwS zez(gCb#_71I{o^K8>fe(rB*`#nRUAwG&kKGtkJJehLb=Xnq~DRbMN?!qNYAl|XQ!O3 z^tGcn>N-sBycs7I`f^4YEVC8!C(_j6&f8Di}f^~cas8OrL zZ*NXxwguru>cXs^s06>OB*zJZO-QHVj(xqX*N{MpWam?=+UlH!Y;0Io|K~*P4S}!G1!fmkyp= zmHuq|+pE3V8+}|FLMmoruAa(@bLi@ERP8CK#i7iM;alIB?WJ z;G7*StxHXS6DAq?tm9T|9K5Q#{#6XNQO2}Q<|QvcX(!;73M8A?f$Kat&nSFAy<(4R z_nC?P%8vafgKQD?Q92x=yv$Nmj_|BP8BT`lt+N}PHQyQTsf&Qr8E#?JT~~SLfOhnS z@57zul0{rvm>fr?uhScHh@a7V0LQ*5y5O`V2tm>|Qc8T=F7Q~k*7q{`yn}DzdwVJFTC@7^!|_hrNn8y5r(`h?fSREQd}vSQ0tF@zFg#4INj${Swy`$@|zJ8{}VKJYDzS>y9O~!3MYJT%<`xZk@HO^*-DxOM~vFgFjwbF8x_Sd6Zh)y)|mfuZqIn z>EwZ_iq1MCzL|(lqzA;hLct{JDTcDF;O2phT_@9XABo>MhdiK4>SH1a_+ZZ zWw5-P3n5#SZk-S9pcH7rzf<(ZSRTjVh~LSnq`Mr@o8mQlYjec>{-`^O`anBoLokB0 zSdL%O%u+Sql{O1dzoYKFzI7YoL#%l{(@$VRv;Hli6VkJ(Lmltwqqt?yUkN>?IP3JWo;K2vA-kt(_k{V4ZBD`_&QDXPsL!aJE3e?s=Gywi(%OAF#w20sP0Nqteqt!K_D1FJPe}gx$C`U8dBs*=-88UJv`2 z@vsp2qWHK^u!@)Ld|cH3(TnR0+My)2uF+_PJ%^)KfsD&prQm^hMBN-b)~EU*yH$ue zjq%L<9-v$XQrH0H^80`KKc38eHK~0 z(KTLk82-ML1c^70)Rzk5Rcnsc9`K<`+3~=|D;J8?ANP~AU>J^r?U++jig3bpa^SEt zQ2`fFOpHm;%qR56xajw_kD6@@h~`fW_7M%Qqhbg;Qa^tk2vQ^Ju4t;!6#lKoYrN6& zL9q*)m#QMxeh-DpexNI3?BHAJyW%ev-9g^J=E_b8TbN3ulB&=7MnCXu6cyZpJ+;V# zvb*3SgEOWNzo}JYXif8uy=x6pYbHw+vQHFV&CmAzD1dM>ys#1J27>R{J6|gehlq{q z|0O(;_m0>nV{5r2v)~pGYHX=eY^PZ8iNaR_A`A;#`6DX$?V2SL+s@6I{x9@`6eiq#iwb+&BV z0INu~xg4L{Z(rC+#P|@^{&5v;UArDT@{v_ocaO^1`%CKnwhF<{gK~T!`X1@f@sUic z|Aj^7Yo{AhFgQjUI7KS&AxZ?aMd^!40+Mp$`&~N$r6oJ7do_c>+1Bo1V~~dptKO$` zG{Se%nVBuB3*Mn-2tm0$RCLTM_j6Bz3?uYZZ2KSQLIOk z<F}9Au$P1Re$73viZ)(UdwVhn48qkUtP3rX zrZa<&!k#mIa+ZxKNrTr_WFq`%BEsgB6#*%NGKWmX-3w$}@oL}w_>3z}laA$KgsU#7 zdaD}Wma5#Cl66DeR2n%`m*G1}(9#8}8YVqcLfy-}54gd>LlSdkweM>$u@i3JSMs(<1jo%w8&y&$)MNw81UY^t_NDiJ2;?zru_fNgntEq zMiFp#-fn$shrs&@h=YQhUWxz}MWvM4dsac3WooQo6>B7O_iH|D(IfJN zV|o6sYCcGxeK2$cCLx|6mldF2V-tfNpn?yx^-Hk``!TvE}!+@-UZc~=ukNz6)vbq^BVjfzr zAZe=+CBH;Voi~%pFy!*#Iuii%U4jXPn~?5~M%`A-DSWxV^D6^4X`G|_&Y!KxA~eZ0?k`bP*weP`X}R|X37-X*PkT}sIN0>cXU zHP+zScG)RtzP`K132I|^#3tpmMUvP?5N}#-ZgQi={}yWFrS~N-(D%e!Gt9L`Fk)XS zuF@>LJBpW=;zyE9may+OTQVDuHp?4a5o$CN$H}J;)=T=KWWfj7Hbk$=^PxqS?i@z> z8dJO7Og7-WIj_|>QB(wY$6Yu3De<2wLF%flfi?TB#-1uXJPYQAS!HYu9MlQy=2^BFU4|T?2)6@_4-k9@D|L@mYY9jSpcV+FIY2pFEo)HHqvDmVzY|K1gQ1cf8S+c~z}u%W%u;zQJZT z-)5W;gVcdncd~Cg-`%>fH$~yF-$WuU=^tw`SyRx0wM(*~4}FibMWIjB^gRqKNYMwb z`01-FFBh2Bo8pSEwR^P1hul~9p5$R{H{ke`*pijsY-u9T1LtqKd~|=+g%IQlx5>Rb z>E_v({%#-ORgEtFrrpd}1bQt(E~>sw+V_NTvN2dG&(Re;>x)v2IIOT^CQ_ay z`4Xyx0JNpPNhaihD;|uw6)hAd3YhTKE_SW%S|cM#fJEvQSRHIPL|WCAi!<@aBg6r}xge15U)rAG#knxczAdgI!b|}gzwiRg##tL6;oJgoX zP@C+2F(_E@@4Rmb%7DIbsCJ;X@SB_nzeX{a;X_Ff9^XQImllm25vr>`W{6CRPlS!c zPh}8tE>{=UUk`k62rtjCHlqVUY&u#N?GX?3;q9zl#WU^8o2?REaCw8?lU<`EBVo{W3sy0-?3xi>1k^1V{nsC%?QR*bLOyi&eLqBsx|<|s;%N>F<0;hTcmCa9 zjAU9poS$!n0Y*tSnAi|-4BF@jr6trJ^y{G-OZs&Ee z{i@i7=|)f<_kk>!4ZhjdAjG+rN9<%x6>Nq+nfmUN_5=Zfkz~-FvvQq)(Uum+>n7j7 z-h$Oz41%uXj%*}eWJHoc-z|i`yTr~Ux27l;x3lhwI?8GV5@X8={Fx{2tVh|4 z0GG=u=^vE%Bg27HCof%P5SwaaX>4A|w)@0QM*R1*1>X&)k9bbqCm(v7v$ldymG?gj zy6RNZz5CKv6^E6rn=~Diw|f8b#HR3~Qh$4Yb_0LXxNqvq-@f==t)G^b0nHv*ddsG5 zXr{4-=b_?ks@L4MoI>9W`DZhw&2?y#=O=T{^X9tNxymgt+B1p6Mt@(KkfiSkVK!wx z`ZP4cE4H*wZX_Q(YAH;mT)ncE{k6*)#H$HCZ|)ZN?T?2+tx)GJ!{9aEqN4yb&NfXYu}T*7NK&(~RIrKg_net{_>nZq8A z#zVYl7hCrKs9nzw{*cI)(@2?bl8!AVg4cNp_F!6!XzFLklgB0vo5*6eP`30 zCezPhvKEPAXOLb#cjz#&RXvI|P<5gkgKI82p3g)ytRitnTE?}5DR8{tE~F;la(VZ< zmq?*qzvlstd)Ve?kYv?haZRf=v`;@Rv$Lz1_kH2A|C5vyhAc5=iv8h_SbMcNSP zVp>jl1;QadD;V~vZl$f_h9B|J3w;rn;p`fVoO9PlZEB6PHtK}c-y;i1S#-x2tw!}X zkP+qzM93ARTITz+1mp`Uq`*a?%r=k;sYoNSm{iAJt4|V^x-)xvV1i@G_{XtUc$r6M|Q^DSO)ZmUwn7y zjrIsWLeHl~dJp1Sqt2z$cIII~Fiu+$uQ7C~JH!N-NWC&rWL|~x+$}0ZaCx#5cJ~tJ zZ0sc3m|9dfB$s_N8!FTC`qD^oWaa zdzXyuN?OuAQCpi))iAF@3#@cWBlKOC(y$kuQNBn*YN5u2PhaZs7RmI?#bRoF>RetC zX>oHb)}p8Nq8p=e`~ao9)-)W-)K~Y*gNTIrKV@vKDGkcsEb$N8!4A8;=N1IvBZ<26 z)6Prv1lnFMvdUEpFmpnSGkcp#xqrQf^_b47=tcjkIWM2_L0w>DD>y`}HhLI(Ac&Yg zLeqKtDK@8Yhmf2_;8K$SsND2mzv(^{2wCw9r@?rO^1W5a1PZEO0?hw6+f`KcU(PSV z(mrTcva`Tc?P8zzm%b9Vzqk+W&rlS^OxGuWROV+#JNt2#y^*&F=VY?bXRjUyJo{w8 zmKG9je!}86ZLT}@Kob1McL;#5v7bqm{Fh|Hc<*TtV>>py1QE9%bdS{;`SbhbuE-7hg{K~>2Prml#e8zG5c53Pnk4Ll zIc?BxREDTPfS4}(k7`+y1{=jvZGKKO61#63e>qjH@#%X)Nb&`ud(=zR zVk6Frlhs#%VdMQEAXw>SKX!;aH|NE+>;hlg&v)uxm5T1kH#YnZ0A|O`qu-x-jPBR8 z4w*~i&U;YI5O9Wr6rw{RQlpOJ`p9RbSX|G}_8t{Z`^loDvWul0hgIL!Csv=+Z7lol zY{t{2?vnh<7!RAgPCBJ=h0g`iuP=chH6a^{M@u@XKh1>b*jq4vyMfc0?M|@HQFGj= zd^oTe1^S!+j<~braXI;!n<<~ra;0|bCG&+eYP$L^Cysz&(bsLpZ4dq3cZM$D{#X4r zhKn|L-9OD;BNm*j6J5CnrMncKe0sA% zVz^nNj8c9ji1%b!r+$yvQr2wIdZ!Bj5v*6^&gzE*$)RO^JC7J9(^v22a2czxDx1Sq5rWL zyLucw7o|0<;L&9A4t-1rzDGNj1}}t%1Zirnh5;ItKG|?-@UB%CiLMm~hZ=0?ymdam zGxEG=BiWLy1W$p^NgyrQb~fCi9Bw;9tg@1%=;VHrLV4(#C&ee1Uv$5;bc(>@Q)& zM}_!+^#DKJTSi*m$(a>Ox)P2bLvs%43#_JVGDk0bOrJog&cTBBq$qr4y%>S2X|D=d z$ZfRXwXmb{%>@&aAr@7=8SS=GL0RZoHu_YBrX)@yBQciz6JJe89ZfO^7;4Zz3vkBp zdNGCN%8TpV0Z8!nRQd@42tw&mZ2uttk?SpUdqCX)>*-a4-9`(v^Y81Uj3j#AM3qkH z^lGs2>gT<(-6Vxia-SlJ6C=jORoEIHAsS|4t0`GWHe-%tQqhq_@CP_%14sq@Pfi^cvl3WeVrxlWK zNCQsJp0mgU@`1{WJ^A}xRsr^jqweFwzOV*-xxSCTuEW=I zTWvn@7C>JT?UQjNedV+%TI%wyuhK?`oRJFMKCyU}U&p5* zSmtM2IuqK>MWgf>q4TP)=k!#TE%J1*I_$beu&z6FGP2#y{YEoa6q6+wiFh;=9}&3F zJ_PIR)b38m$JlaildSX%bs2F{M~SD{zS^&no3bRzcg)QkiHXpXhTpfL&7L3lG*^L* zbag~Oh1oxZMaH^wVC`5w<$<=J%#iAxrJ{8@3HsWxhGd@`i4ftJ)z4F5H}LKVWKijeH}oMY)<%R51#D>KSA`ZCq_+@&*;s{e znFbp%MholCA9+wo2P{vqm+icRA&xgh_HuV{q*<)+7u$M~+Ou@am>VlfU}dGyqPLxF zOpHjk388(XGHnX?W`a5^)yr{PL%m&LgJWf_ZKH5ayjQDzZ~M=#Lt?~rqdO~c45yGq z(ty~}oBU0xA_hIGtgpufS7wexvBEF&qQPW&1a*{OF=a`=*}t~BrlrpU*BYsT$sW?3)2D+ATfB@8Xb`#IO7CEA#vDnfw7tPJ=%cP5PcM#ZI6B+B!m!D z(ml>%>mo7r8|Jz~z2r3_K~#`g(@&Wq+HGu8d2oAdhSu^L##^0{lYu%NX;aqmf+Qd- zFT`T&H%N{WBHdlmF?7!i1K%2-=Y7v}&iVfMzU!R1 zF81uQBo4cx^`#p=^F2YL!k#QTj5@* zp;F1O2Wk05pI3`-t-W}q-nX$eE7WS<;x?%&&~CuYNG-jjJd+mxdFp_<@{3w9K?XpUEM4pHdu5egG@+ zoSM8U@Hy`UYo!P}y2RZiCJs0BFnV-25&c@>A@i2_UQ*c%`@Oz4W3%#!`)t*FH*Oac zzuVpoob6L!_uracT{@9DP+4<3$3@xpNffvW@>(?~i5*^Y2^5QP`vRK`FS9#z)KVSg zM(^Z(fIrU8!gx~<4ZPMTG?c7DGgqnlh%DL5GDH2pZKO@XnvUoA1Fe=;UX1r$(}u9i z^k;ND5NtgH9QP64dxwPvn*~!R;uyXduR(ao);gPVmm=VDqC#<)tHC0X&tLew*s+zc zChpg~(`tJ8=dN%aF<8{#QMC4%+j=8k^1}7K8U-x$j<~@HHjqkAN^-csoGfCuYMO2I zb_DwgCMYl{B=g-7Kbyzjo7wRN+o^-KCb9vqiQNg(dB}{l#9)?5BQ`-VIaZ*cOM`$W zHdeM`Ecu?3QPfLv@i@ZOT8{M-C33F&?(9PPFWgwWm7?ytV%`XZCA@dc-9p@&f8g~w z^Ab9o@3kNLOM)JeWsxwAY8-4eOGUu%=hO5jk0EJe%fjWkKY{~-<}mmB1D%K1wsJeg zoQPj8ClwMaeQXc!{b@<;Y-07oUG7Euc)KB+0N#giQb}u}!7t&C8;ifY6StgpyM+mR zQv}mwQ;n)3iz@FuZKYKja#-&8!fCHVd5)Q<^r>9@%E;VhytCB`7n=?-lV4MzO?Ak$ z+DtYu?|V6rVv%&nr#c$mo{*9EOO|W?ci9W_Vyh@ZapHlXJd|xh2AXK-2x_R_@|-Yj zSh8R!w1h&fv`vgccaz)UyWnM|(~Gb1>4hz9>m^1nmMi*we9tPv8_AD{&M&vt2;*8k zOpy$EO*#dE2ZCiqT)YiF*zFYbC+V-sd}g^&unP+(HD13N{kvb->nv3~)#0i*CR6&H zh32aMYY)=ZjaTy>m4Y5#I?2|9%5itxZts+}(iz(6#gQ8`uH;hpX2mm6wY^W@|^YV5j_|VC=BmFYoAty4k2(WL6^~ ze0Hf4)RO2F=qaTZY4nSPYZS}OM{FsP3Z57UT{3J?sHI(xPUQvZ&Dm1saTzZ;M_;Fg`sfkj<50W&0oji zHf-ZJ)JC4HpRw3X)h>RhoT@Q4>|*nu$2|HxvV8|x+Ix5tyG}~`uBXFF`mKZpzVgQ- zSI?Y&@c2}IAy+P(WZJp`CSF0QwtwKIZyt2?5-y!OOn6WxGc4^Oyyhw!+o?qdLPbqo zzuH)U6>Y18OAjnq+-S75&Sbj0D`pYT2=sKvtTPgJ-kD?g5 zpxUh1*%(%WZ8dLwnliMl^eTSTjD+gpH&d7f5J>#`zR*O{z-db+7cPtgHX68YP5!Zf z!I0X6?a!uOH?Q?DZRcmEv;^WCgUzR8<>jOEh@>jZQ7A8;v%PkwsGT}QaSM0fw@gQA z`CA`WR?O(=zGWIk?t;3Aw_fkV!NweA-#i?D5#FD+ti;CtQ$bG$dVl`>^lkY!TchvA z7g|tYrW)ylk;)?F$g|lOcpEoX^}4(No}Q1-8%8c^_d{Nc@z>k@B){ucZ~fLwO9JtFY+&@q-G|$g%kf&X z3T4I9EciNQ3uP3h96V4%GdYgl9Z&+=;41O*LO z_!QDP#P~W`R(RUb>p_5!ePu6+B!iWb$Bjg8II&K>%XkI_@f4&$bjH^Cz7(B~h|lz1x*ZZOpiUkkzdryyNY+e^k-S{3o?w$7nP~^m@q?C;9jzdn^+b^#vBY0Ss`bZKfC}HNf zeB1bTRNp$}_-Msx3GX`!?;a(RZg1a#<8OoBExinAVu#tumGq-(V}dX?`Wg`!&9)VclHB zcLfhhH?u@7MPd3})K1{$6^7N}nCo4*NhU*at_5--bsE{GGq>|}n5z2S3MHGY7gonu@af*z zO*XE7KWL*(Rf}4dQcg>WV0F?fm-WIyIOwGPn(v_DE#Qf0pu`Uz+E*Bgznca9ir5(3 zXvxlYj&Z+7Mj`kU<5$(0=VaCXK^5-hp%K#c5F<4{u3WjS6}Epr+CS{tr7)tGBt)W= zdqD}~0UqOle~#^CYvYu5;$?btJ{xU9McWf!Wbh#nL9JmI4C-jG(c4|S$){E!kxN=h z47v1v?eiu79tQhl^4+JMNt=S@KV-S}UVG-YCs2n3zSpB3v6s+(d*h_p^s`mYa7_55 z4;|kibDo-?K1+#T{qSw?ucXg-^3(#QN?sK?Wn7r6=MxOzr81h6{<eDo6^P zGgT;JcvL`1`$+UomRm1*Ud1XYB0S1Lw*y>UhKQr*{|0XJTqBk-unhg^Oh@i`;wA%kIBM0a>>0``#c{0avLhF zNj_5&$CdRtaoZ!Svk&(}R7qT#6>|t*frfNB7qeVXC{JEw8r@N~QtY9)0ZCscN4G=J z8wXY95cPaZW-l){R&fDQUNx~b$AIZqPm2yQzLCrF(3$66^nNT?)Wo86(t64vR`LOb zB!i{M`6#`{0D;7w*1Smty2d8isJ`(ylHU>rXP?t=;2dYbABY|@bv03uA;hwNo`pb1 zJKx*K{W{i1XDe074VJ#B<0+1`i+Xuva8azsNlyc75z{^C8c!$vDC38t-RjFI9jCcQ zhsS;W*Jy)F$6S-@{{FFivDy#1w=0m&H{OiW$8yrqbV)32GDHLCa~%{L@;k>zbOmzK z2V|}$@9#{V{Q9}&eNlIU^m7X<6hC8|fL@G6h>t-ulw;8DHPM22?VTEdrtfEWqAv}a zDt|5a#YV8?RRBajBrJ;wq@+ z7=Nr?FdLU0Vh~kQT@;mvq=7Dnm|Pr*VEE<}=pu=u+vxn=K2 zRn}v2KsAW@Y8htRBqY$dTk8wM;5EA`Ocy0c@O zKVFD@5v8dZn-vTFXg+1{3-x;?r|4PyTL;P7?iZj_w2{$r#fn~55p}z@*8sn(iCLH~ zrTS^!2%Wvcww~KNpIUFTUnOJaEqtD@=gT9~gd1P*))q@TGu_Bn{Unor?(rte&T2VT zeXPO&mF>fjo23xR8tyH)^SS80ZYSy?qr9(RAPXx6cA>*X39psW~|5bi_H+!VpiqEAhui#nw1}DyMU$oluz$gom0^m9LqPh((8K zs*J)~V2N9FwIk~ckb%6Rqu=3GDVO(b64)SJ;v{|h;q;bj^Um3+N_9S|V`^f`qRPH= zu^O9=Q6iL<5gHSrw?0dLjyKaSVWfk1#+5F5w114;U+%T!Tt7i3&3?|-8_TyEhkYMZ zkKKw|o?y^UD^gXgmQ~WJW-a-qPqL-UjOWe~Cwk4h$LD@DbEisGSd{Uj z;y*%hmp?mLPg66vQB#&>Qyki9)ODojJ1;dB?Q|?OhM1afD|LL0pe`EJ>G)jue2}S8 ztsB3bub98cvL6XmM<*X|GG>C1)b zJCZ$eC5K3PHeru{RJf5wE~$RllrLgdsu-e!+N0ciA3}6^6-AP8>b&yvoK*W!JE|G0 zEnE#ou7Z|5Zph=BBQDIzopN4t_=)ML9i9(TO)Vgy>YHRCqksUPH;sgQ-*6X;XhS zQ9hUurQx86qsEw1c(qC^|L~36q+jSTpwW<~E3wk-CKii)Kvb6gfTP{D_4?KozAAh_ zjI`**6Qf~A92~WG5UJG zN#*_I6lUvc@2uzVuD8MG_UMAuY6_OdwwMQPW5|ArSRxstDmRUX7zR0ZAA(J4|^2k;NmG3s~-v83pIrxvuHUr~G;I_^coTy9Dwt79c|HbkkA zLn-!M@W&`UNYr`469~MKI;~+Q2WN%$j;iFw6$bZnRiTRk#kk@*>WO-JD5Ef)nwvmk z`Lw0i>$k;kXJ~k(mIN4;-8|xcX(F|&NzPa+__|{gADMN2si2B5fkayk8kCDrWCiFk z$(y_l?ZRk=*+jEIe$aWJS9M?8nvEUGt%+srbIV)lh|4dRG+T0y&OUhNaKClCI2X~){@%dcAs}e1ayEri{lh1ebvMXxW zpSyyMtj{?ve>(kxcD2=g3XPw;Hz+qPJ(xUB~lna*$6?r+&YT0+n9|2l@Vt82(9ian)6LM5ZA zzG=if^nzAp4u6ee&=^$h7F(GX=Q3SB-y+mCH-tVYQditp^7*MZKGdqDxa%TbFOIz9 z4LKhbHt9X9H<3nd{0_V~ELR%K-*NxwoL;}e`J-rqYEsA2w3?}$o&5y8h6GBXAmxRV zYit3%Fib+Wt42gr(bvi$f!uZ6RkeR#LHiUV z6h2j_SfRKCeM`100W z`1Ewvgg9#=cJNMh8V@K>o6p3PlV3$G6 z{dG&~!!oIA`jXh^*UTZL=0!G;0&4kNI=`;xR@*q1n-r+&S8p^9#}q+aO}q^{Br_R3 zRkM;RIcrwZeyILl>U2Y5o?ZTWP+=2`EN@G+T~Umai&FWLPp2_Kd`N>JcFNz3GPZ(K z>zBI+4=vs6*l~y^pV_@iR_+_ya5D4|=>ut6I*-K&!GFsp!gpjXQ}=UkL%j6s==+2c zO!5&AaR|68+l>yP07yyq)4%V?OLEd(R6hA zWM@5_PROPf#gaw^e6%E}K@A?qKR*P?qq|mgUH->>yxfeCObC0=gtt=lOZZtRCkY@S7NrfOvR*qm;j}PyFHtytx{)qFK>N^+=qK~&Q zO3*r&JRbYm2Cem1u<&O&(q6M(M;nRW7~$zLwwnjnYuI@iObOAly}D6S31jY$f|vIv zAELD?qoa&e=vq&XfVdl9oT2rZ&bbM_F=qs(Pawy?N!xc71eyNMEbC z9eLW_#2MUrt@ULubrmddG2Ci|9)47|FpOY87F{QmpYAD4v5K-x9ud=8=1#zLM6}^k zJ~$GU3OQltJlRHAmG4-lK7^E1yP7T>iZ3tL7G!@XoJe@3n3hr!3$yC#w;~&?(D^|f zO`JKQNtjO5nEv%+n)QU!U9H)bs-u)9Uku7jrw}dEZEY@mb6F(D%V&0?P)qG`*ZG?KG0yzvCd)<>co>xDbl=cq7OU^a4v*5Ycp z&2}YOf%&gHYTI$?{O7F5-ZxjfQHfj4x+{kt&9oJR`)PJwv5AhT`-vhFa=1KunUNO9 z%+WUrHSMN2!WBq8*(x$_ZyP*Ekvp!rDvJ{?M^<}p;eDqJeh}>dZ@GPfoR@9&Xcaax zB3&?~`s=8CnPRF-zuDZ)t@_@*d%yOVyG)zk-{b<@ohEV`e`o-Lqgw0a0UsP08=(%? z2b*H8@WOKG85xr~>MFfLy`QT9_o$e!o=0OnR(#}Zn5o@1+3&VBX&N^bo0!N1wq4Ib zg*N3*J#l@qG*j>1Rc$-faG+VFGcwI*Gyd4q(~h1l^241+ooS-pCSVocSxZzBYVr}k zUCi|K6=>)a{p&>z6{cNppcrO-P5l+cWWJ7b3~tF1!hC4eZnj}Um`zkvKmS#UUCuh< z+s0^-bsTiAPa{v2K2PP#2hWm5?-Q@wu{^c*X~e2Sc6K%dxa%JC{=S8L{rYNccf?r9 ziBf@9S>|xQMpvPXemXnMH{jO&?mCz3oSdYw{!@z-y*fRV&_mjIu~+f7?O_z<&a1tu z#wA@@2$|I(q<)>93k5O+2H9$bZpvFR)=};s;QRb=`#Mf)%ZRFbi0fB&GNx3?b{f(M zcgx)QGNfPE!#6)2#fA_r@)nXB##JwASTgts|cAeOIn`+%Zu+Pbw9w z@MaZczYsvGO2GX+5bg682sc9Ta6$vekOxx(K6E zd9~3Q3wb4{Gx}qP*ae-iqJUS2@2S3_Vn()LNtF>>(Y*f1YTseNEpsk7z)X9m%@f_e^Bf1^$lC+yr_x4LdtD!)CSApB zP7?2kvvgbR1}i+3J)f>VpI9Lm`FYja{d`+ut1%1K`g>b3doJ>536mJI93ri2icY)I zUV^0}&cylU>L21lqkU-{A}Fa`;92O`zO7b)QlXGm(ESLQqH6N5pgP(t-4`_pY$b4ywQF^<_FZ*ib~>JF{jthac4rO>W6A;tAJB;~ ziZH#7xC+NCspv)3Jb9@|ul1r}qaucO zrK%+Ly%(jN$fEf1n{M4S)PBY>^_Q}Sfly2+1-HaDqif0J%+6kR){bNsR~omoS*IdP zGvZ*wJ79W8-j;?~SMj#Td6bxL7RujrBBvL#I32`skYNND=YZy%*D1}WR!Q*L&Cy=e zZffO0G5z8Tndw303&+H(J;|g8vBg~(?M^eVs5Z31$|Hk-<0*RvrwtAGZ-d1uR2}Fo zS~eXI76NL<>j@B{Xy!=4^JMqCL(^{=nKq`_AKZ$Fd+*s2q9W4$FSj|u6;o}q$72t< zoH~Yys?JJ?Y2#`&GSFQl>n_WptL`fx%{#x+C5v>S)cr%EBM5pM^q z)FO47250+Vh1OjP| zqhI!t$u&0&1go&Fuqnfk4ycF$tyzmUv1#9^B6QF;+5E{x^Tx@@BSM$-=DD-T%`4aU zeIpb2b-MH>QH5I6(kO}%StY^pH)qvm^*q1i@J}&ojF+D+NpA>C060sozD+FVx<*~) zb!3%3av2AQDAhN(L~?Zf4$YN({D{Js!Ydlq$FZtaISX&{dA)wH&Pwld2l`y%3_bFA zF-m5L&-6giwAP2I12R}=_nUOjUor#Gk2syyR9q_P50R&OWosvGYE=Ezrbn|XtKEs*Js-d7))do4S&ihkc@e(6%2|fDSBlSBhR@05x7Q!? zawcCY$4ar`#ws`ICx|xm!B)!b#BIxa8!z`!wV8Q^@!>EM8YV(?unp~Jm_G(rpy_iV z`a^Z9pVWeL8yEHHvDoWOR_>RxBp4W5ePynt5m7 zSygkDDAtE^l{2WhFU}5>k*netmgj!Gf~&Hb{Tx}qkrE-={PF+gp{a~ieHT>>@<-pJ z6Xicbv>AFn2*>zT61#UfA;-ih`7O}sMX;+1+<3rx}2|BO*B32fm2Gn94yMCRl znOP25GguwXHv)q`kzegiUG5d$iIJKY2rJaBQQ84H^yw(# zZ8a;^)C~!hG}M1|nEyd|>*>EAso`^pMY_}{NCPDv(SPNfAfJuBYh_Oxe!s=*#vHB> z^+-3e=~m0OeZ0T?i#9yw$2;6Eu!+0%xf0#AYi}+jZkfLO7ykV_n-G`K>LmL-Ij%3x zP6EX`V*k9^i0rfRvQ{+pr0p+VWIL{hBwu@;Fm>PgR=8qY-YP=FJ;Y8jKZ|4Bt zeH07&azBXJ5dDXj1;@Wn#z~^J?l4(&?N5;4f9LzR9AUe~$eXIP(sa}$?QdFFQ2Cls zXj3MPNR(mE?M&{_qm}kw!hi*B%5TZ1M8odB3@@#+pEGdL=uPGiueu5D9Rq5t-FQ;l zuHliida1m=gbD`##93o}>q`UP{bi4dqb0mZ3Uh*UL1?)`>Gq#T4+N9IFX2qx7!Q1F z%qoEUPG3L?CuWz zpWe_6+%UV<>k@${Ww;C%7nfEQ$)Cn!swC*Vv`j22%V> znEs}bupe>1^Ibm9`27H{_s46}-ht<6oXz&8$-kBGp9-mTc^ULZ;^+N?1LMXUPu~7# z;r=7yI5xDescwap{U*QdHcw7biT~Yqvta8lL-WsLboyhY3!M+Jx&M4=g}qn)=Vx%< z<(VS&AYu0M)>I9te(b?YPsmXw_A5b=VHCn2+ZQ8&ND*A=#|l5rGjYNI2T?-oS^Lux zM%8)lOO^}k+cWh&qMQG=c>j0`oNB^S{A=x64tKBDJA~bx#C&M||T(s*Uh4Ghw7;wOaWA0< zw)OvL{ARRyM^%&qI=JlO>P;F^(V1y)5lWVa0p-~hKxH0P6aI&R0QO@19{yHJE$qto zINwY~a2Lsc-i7o}CiVY*mx|#`Yx?LssFO{lgVW${Srk%Cw8m#$p@wK-8CpCZoB|7sg>F_t6wu~{G# zjb>0>`uWH{CM6|>D80gC&xlOKqZ9<9@`FP|%WGf4yr3vtw!PSom{IBHl50+tBOF@8P5#T=-IV2{=gK3^h&XHwjaD>0!o4M?>CyE=9u*Ucy zyeUzCjd&tJEC?g88ZO#fvDJ3eI=<*-iOy& zXBYq>1cGhOPWNWDT(+kt1W%@&Qz=EgJdB+|?INl7UC!V%KxZT#6{F)8!-RXWBQjKD z4`l?ps{lV;2nm)6O?7C#ISU(K8~73|bDPI)pUC|DXZnqSS2jC0T5Ge zyBFbNGol2(J?+%R)(b$~n$2uO?eP0PVBnKm`4vf!n-zTxL+PF?WCsm=wv!20m7ygK z`yV|EZKtZ001uQ{Z9gX#J^guTr+%l2Ncr1~F!J?OKKogbXmyVysB!{!9kz6cP?@CT4zUxV~$el#?s0PNm3KwvU4qma&l6Y z61f6AS{=LxY!^bgdv}v}JC0SVDJq{4M3?VPJNJ3|@st4`P9mJmpkdrwgn^-q*Jk`F zaqeom->mQ_+p9OIFs-eU>G#k3-CtO)bcsK9 zmyyqJuZ8DyJ`DZ<9|`2;zeA*KvOc>+3PpC4#u}?<_y>+y;l;=lluI| z^xZvk38TjctO_26gM@veN2@$Pn`8F6OQ`?4XCg-0#9yaQ_!=Ht5O#O2ix-UYpKkDi<2Z z?g-m!BVB2N!hHjBfH&IVnLBDSZoiwzYaQnacH?OZNORsSYnQ9C8ZkNA7%8_H$TZq< zY`)2eMc{a%xee{rKtWc(WshbpTj|`QqVAF_0hteyxjpTR-YPfmqa@=tQwTmNg@N)O zsX2V{v)5ro<|&RJcr5z;6SOvF+~;CbQpm}kzV;WSaw_|TGY1+^5=DQ`*^0Z2`TDvS(~7r&w>CEOiwu!Ur(7e6d;9Chj|I-xnli#qi}$P zXQ%C_zo}H)OsKAOX&Q{7=L4w>EGUoAJMd%^0maUq?rv5dIK!K|l0|gNv)LB@=anMb zwNBZ!_QPKz-5+uWfBI*Vk7GgQr&(W`?dS!C`!qHtVs{j)4lh3f*5g;~D+4Cl7oib* zi|t*#slvgK&}w{FP>9%dyfw9q^#=vTb#iv~TYUDo6QC^DreVL6W#zmXkAsf#*;Nog zwDUee^**@?^E_Rszeav*&KW28jc+Clk%|Z&kV*eVdHCEY#MxKaG0hgYnDHn0BYDwm zep#!+>{D$C2x4pDN29vgGU1dAgim>SHU2Q4)&VS1=8cF8Wnlhm7e2BmsG6q=xqd7< z$aGBNv5X>z9S-b+@GSw3f0P2uCK;biLW)x-Q%-)iFr0zVl1SdO+;WHk30#gs=H}<; z?;3`JAroX`KHv0Q2BZgbm8lL8zkvTppvxap7Ekqvp`Mvr^8yKkS_muaohg}#y7ipo zrP|dr?wcRt^_+rvtVavA-coZ0H;G@I4NW6B)mbtSZu`LIbiTtEvp9yDr+MZ8qd*k> zz^{Yfc9IA9oqbIX_kYX6=bg~ZwXPPbCf#R`v6_ARkk}YLTbrg;y++(d2_q18=(Xnn z&gJ`&6xva=On^v?_&bn;ZdkTltOSK!bQ$@TNr-8d0M~N zSX`brPlqu8R-5}KQt0Y;%iL76d>cgRg5TL5E+>Z%$}X1%%GHnoh1B)wP%IFbQ1kYa@9Jl1^%9Ku`7lQrQlPk^Qb?vT8E*tCj6gokMz zFL5tsjAKJdmB~=G-c31}3Oul>cMu)$1-gPo)oNc}+*uzhNv{=06jm5D3M_xnyg3A2 zq43@;^9?lFHUAYs&44sn>(98uhd0UYZf?=$Q>EspB5YkGeQNfy<|aCrG^l z-XSlc#${c#7ExSpa=13YkY{_cJ=6Ln)CTAua1@sLc=+T3Prt5~!=(DYgo{}*b0RpO zpX}*$XtNNVY>ae)NnhWu;1fi1EeV9Vryl`Dl>--`r=^t)6W$g$kd7WSadj7i+Ax|L z1GdCB(3z3)nF&^Ls`dk$xX%FQ$UL>X6xe)1{tU2r4;~%%bl!#02>0U#i}oV7T0N`v zzol7-1HFk~Fn_q9;N?|UJ6B;IPPTwr3>RlV{Fdy$+#C3-$elt?f2{s4aly#6obGCq zqpJI~V_@wy6nW7a5{S7c)zB?d=~a1`IiG`1dX8=PRf}|ZqrZKD2sZ&K06>pRGT1s+ z2V=TnTQg%8HBlKwx%VWQ=mKT^rJd57^)qrp%m{%(l*E_7&vly4_T_hYjEG=%yf%u5 z!BO7fbj(}38&8ju8bo~0JOuYzi7~x03P!nr1YGe2 zuC4XUZOiPrJ~4D*zwP>9_DWe>2x&5$lugIv+zcI5)RWBb9et?ZN{5=J!nFZv**b3$ zqshd;U}_E0@m!XJbO#rB?BBlch}h8nrI;2-fQf%(a^Pcq*tP%EH0ktTzQC}*b3Q-) z;pt`rG}8cRCK+hjXY;-^Re-4y#r=FUIt@E1W*e2$#W;29T3z=$7?aV;V#qA5{T$&Q z_eR9c%>fxITgr>WA!R(#Rp8qAA&e*%ILN4xc!vXQl}MrO8M*~5;Zo(0QnYDTT!$pS zD4pL0dQLtiKB2W2Frjz=4tPkVZ*#l&sUGrK4L>aecUim5Q#(v zq#f*caU{1-R#~sw%r(s@Ox4(@4^@~ebHl(Qe{Hk5pbaZ!Q}*x~VcXdTPbC0lQe$H7 znNO6PDrI)|^thrJ;j8=+_Nf0$5fiKo~y$fY&M} zxD9j|o+2aVzzOgh!eC3yCw#E|r4A(>ouuF{fS2VJI>!^u(=11^K4eE!W)L$je!hp# zjVY|?rD5Q7n6%7IJi2HlR*n(CN3%Wl0pOnsrildwcVM3H-|mon#`-*kL*g&*nzU^O zu-HH1^t+(nrbC)7V=?1oFd~?qcAoorArwA~HPh=a$sJn>u})%9y~JSVOw{LeSBt}7 zIziOC+PRyv-NU}_{BT9sc5AX~776v<;*H+T3y!b*BDkIv-8~2Ja4(3JbT^N9e`Epp zx55jI#?Ea@k#Wn3@@!9h^pS*{ZB+ytT%Bc3VbbNr0DUu z@rtum=CCxW{?csgj}8kDX-u@B<5sJ z@vYpahrB5O*9+>`xj4>xEPi^ApXXh!HdzEfe_I-e+a;%}+y*N#BSpGnc`q~GTAt|F zySY+&tx=or%r^R_%ZO~Qe?CIRhh_Y$Aaf2w0IoCcRHQm` zZ76@06zWrkEDPAK#~R;rfO+USgWt$d!Gqt2XOKK^3S_( z{yVk!zskw{m;1%h{_he+%d=e^a8^JHZ#SX0o#5=)+tF2n$&AV8;XAe<=vAGy^m{oL zaEwpO#=)TstPLZ;*+lJ*H0P*b=0VOxVfSLsM#sgU(o09!-^Ot00nT*%Ew)$QekKhB zq)Vp|sO-ieh({f7v4BOfcx!}QH(I?MI|IP@sQ?(Tr1)COhTL^nWxocFQQU@K_yQY+ zXNbKT0WbDyQfhMQPzoG>fv3P>ua6cb4;N~4kB0mqc~xAeY-{f4AJ+oS81=ZfH3x@o z*RE1p&NVec_@rzw2J_eOk%fHKar+IUd00d$S}uYM&^ z*=aZk19rxMy_yd7*y2 zCVSJVQ67kM09^yQE4=v>c>=$n_)&$u(PKi3x%njP;>eYrWO;C5MXH6&5OywDysTEa zNpR>x-cGu%^{2ZiN>C&c%Tm&yilxD9&P$q5orl&Sd}jc*EY>(o@aLiB1^`CbAPz7B zF~XGWGD%c>>)yj(RU~^w4iu2f@zAqg zIG{)lw3B$P87wU=olfTiy8u~0r<3M)?&<6|>$w^$l)fmizfn+vMYq}DQKHh)>`?xb zj38E|!iX`(h~=Vy(=r7nk0Gw^E`U)?@%nCwX8<#{t;{rf*JwkFbZhty%uG!?{4ZTG zS?Nynh3ZZP{1n5|hz$oymV;TrmK+p9E)n5ks3%686SdB{p_HPN4aL>Owj9`&VJ8g$ z)Ank^nU_Sj#*Vk+8@3x?KKwJ_tr6Qe{!S1Mi8~ZR!`^#NsKa&oUPN!Mf#>pl=lLxV zt+Dec$1-c&!ZgCrZh%z^uYLNe-p%&l0z~iN3s?t+4GWxpl9|Vxrg>dD{I(dzeIn|0 z_;d(7t1)}GzV}uYwVxgKp24Q)XUH6Q83;3@AA6wIbCc;56u~A{6@4Z@L&+EjdB{1M zPBS5_!|xl6!$n8wb}^un_5mA|m{q$2zc2ti;j;S#P#Wu~{mP3kkhfS^w9W_vE*5~N z0Eq(t3Xm&CI)K;tL%pV?*r2Sj(5}*UYLt!wHQ1MjVM)yesySE!6`yGEtRO2#1D=W> z!>Gu2#Oty9`rsRb@CH|rcEwLR9JlpBX&Obe^Ca%@QBK#ZBwf4Ow%eR(fX@~6KIXZZ z0`8UDaE?MS1OvBLOt|hm3&KE#S}rE0j#kxpd1AV%#&X z9AOKduxv!L2%~0*dFeF7$IOa z{8h5I;DFb1a4F&dJILL_z+{*MzdO5*N5bk*US6I8#8;@^ZHw7a8)QXNK-e>b4MR`L z083yF2Ie`8@6YI2gM5inR~#GAIA9r_0o~IOu9zY)ps+Is_Y3aZ87YWQ#Db~prapk0 zHo8-VYsL>()4Y2{E~4bwnRs9zrZB6UwX0ia*f|6xSwi5CZzvrbH}?&g_OGKneb6tl zwxYMe^FU7Te7+?x*>xVZl5C%H>prsppe;FgE?TZmh+a(w-2|; zjF>5`cwZEiJ_rNU|H$ol{b68*?3h)o=vDkGIkN84>+6uIv#{jfBxbmm^om6$pEgV& zbNt3Foq%!+4cVgYuNo3nGduN zo<+@MmVmgas2!VMk^eFxK+UamW zMqW;C{!-!LL5;0X zuS>;GP9*Z~4j2$o@b#S-fkF=)oO{`lW-NY^?)5Nyim$H$#y2c!NELQZns2iQack{n zd6x+o9w$J3Cex`=D9utS{wh$HKNCX( zv-|PR9jGc?K9Ja*SOT_g-hZx}WjpeOJhWa9zh_D4b5`(8J&<(;J=Z(5^5Vkb5 z6j60H!O%$=|c2?DYG z(~8VbK#Tof-ao&|gM)JeOGI~LfcQecHY=%Q>$eK4OA}!m>wu;NTytt|z~IS^)=JBv zC15D;2niP3&D6E!U@J58V|6;{goUThpc*9x`f0u>OEptfVj|Zj5Gp1yXD4atRMKRl zCwQN$+Q0E}b4Oy>;^CI_gPp0J?n@t{EV6rnX@nNT&%J8rKRi-c9t5cnCH8N>FWWnr z3c!>waE2G7z%G4W9%xliWFF~E7n+n(v7=Y5nKVU*tc57xONBz?t-_SYZLVq5dX8jEY*f5f7sOsPQfL;lSSM1pY`(;jD0 z<{w=1=5DQlPO}lnmq{2z0&^7wia?U7o9!EqG^1bUN){q^H|2;g^LldM(}4|-(Z}o2 z0Ct*NGSWJmxM(2{@)_$06q0x#Amy}aRFVOS1O+9md!is!%2BqUIX7Kq+)lj;&=MS* z?iD*fHBJ^yc!F-Wh_6-<4sah~HbOb^+b(${+@?)flO~#6J86!0dG}+)S3Gsa0-|Sf#l~OG7Q#Z`1IE~)d z44ORw>bsc-snc0kBYrt^D?XboL{odjaGm!q%Fhqcg$d2L#mE4-k9fcHgY=agTgRne zkqm-vfan8sM0yNLkDp{ffKg^pYd3Pf~py*eo^5(jA#L%yf;xD#+c zIN4ia`s~BGOzWr&&FW#fM4(>>KwisAEVBEfEZ%0Jz^n*7#j#cE*k2avmMU<}v94^N zI6$jQyDj+q8k)R&Tq$C?#Ry>UX3s8wr4(B_OCrzTDJlV8d5+Xuub?C!koWQeH1~=1 zXY04?y3XJhx_hr?)S@Aec&$u`bg|F=c<@`=4V)kbY$?Hww*~}*Ry;z(2A1$&LwL?! zk?S8>iX(uHA@R~H$Xkne-xbecK7&Q~Hxe})hM5C(ZmO~vk|;I}cLp1`hq3J;L2`}v z1fUt;-~)2Eal@5!z=rSMeOIJetj7dWz0P@E@zK$=AeFp-UZnA$N+-Jx6dU682)l3V zd7jL;)ARG!^dI(#U+4icBpakF0&)-$GeRl#479UuQ??Aj!(%y$$tKg^92v9+mz#1m z{SPV3^v7m0qOMA2bbW;sBrWMY)Q?YkuYK4afHm+Cl9Bgg00DfMS;c06doYbBvn=A6 zL$!?d%4V|xs|+1|^%*ZjQNZYzpOCjooaKXii5l$-g98%=>yf6UH@`NUL=Nf0D#W;9Ui9XTQ6z8QM zxKuo(AI^_!;E8zl+t;4X&EEjPfC&kP4$HI0c0@DEV{@o&Pa7k0NkO| zRKl%cKSxG!5C;(32sX^9s5p{{0ty0-1`+|0prMMizyN}T9*_|S5i3E!gVLmjCWhVy zDbfQ;Xwrt>s~{zmyN}|0-(Bn8weF8^t$Tmm{LvL5<>Z{b-~GJrj&*2|;I0Nlv@1ar z9E^Qs|8Und@9hV1=@-%*HXAitr@{V2B63$}Q#==FE6{uLm5o@h&T<4|lf^yeZ`%GI zK{6TzO^Z+EsN{fdKoXuJ4zbGz#Kv`bXAfy-aZTB(0mzwzi3q&=BBG;D)sAq{fwZIf zYigj0nOP-(xUn$>V%YiiYwx(G$A-~;3&_vaOE_9~GT`IxU-y(XsJ_}Z{%nMwxkSd{{6=r+1urXP`m3ip#LU*ZO%|K&@=La#T zfW^>{j@lsiiph>}kk|R59ci7e%PYkX^6S^_yD)hFv}duY`KQ-+HGE`4a>GTb8H(sD zKxJ|V!qC2_Xslc4k&kC{_c!9~uF@w#Mf~PJFmuTYiMjzjo)@TV^W5cAzYD`;`&j2- z>gcVqN6%61go)4)QD&W+)8m-e@2wbeTl6T6IY5Db_49t(XtKHWP@ z9OgFJd&&>+b{RN#@kw9H!kn@cHi2*eaTV|RD7M0L#iF4eO!>BJuemDfM8O7$GEmte z9zmE{7x_@7WhQStR~qS=fD2(=`6PPg``d(H1vw{=LeTbJ{opWh^;1>p1ho5JifhTbuI$-3_)rruIG3X^r> zdTO}F^pVO?(y4|L4PsP-1O@#)D1IFhrLJfdL~r|qDEEnCMHeJ*dq7(Kt-&jMTi|R*W^al?VE`^-n5Ny<3;#0zy8w$H*ue^7id}em)rHn``f?0wzaz<9{4|q z96_+>Umgbi>%SW!{ilcj{}%;^qw4Ro@7uBYuhX6XS3W*PLLPYRrvM5xdb~<1ggT`Q z5_%}rpa;0nsj32?2oH|y9B_tfxyIGb7G~)Ovj-04U1pkhqw)`K;fYZJEplH@(7WC; zR*lD6m=wCe&XbDW4Urf92&ekMOF4Ap2p*+JFeWE6!IP?Uf6`M9QnD8==pYC|;=#?&*9m_|SzX2Ot+{u!>`#{L7tw z0xBD7PFc!Lv}M23bMCqiku;$hceJhe-cvxy5wuUvk^->I4$oEE0!+asQ{AOsG92J^ zFd<(o{EJN=@#Glx=bta3U}yn0P@}@b|4f`yJCUsd`6^}Ce8ssQuRL2#5e(tk>hgqV z3{4M2e#a2_@PN@5O7`5Ep}tE?>y3N!#2h0|fH&73ogJ=X|$~8#>B3Id4K{o~a z@RV8gPWnEvYIhXlwmz03K63>*)CbYF$t$S+gntk2SO(Unt=TF%l)h#>kn;5Zzgx@si)y3sJNmAm;s&Lt zpm1s0Pk#Qdd;SQrdzu%BsT(_7HzrdzyJzsfKZ*nk<4dXh=JgQt!cIBM$D|p7>0EZF;%c6fud%zU?dCst^EQ8BpOe@Opr@f?}rRroE$)zP|#zu#RxN`p%5#0^pNq zng;|G6W7GqOe0g(8S5?mUnBsomFJDVI|WrT-PE|w`K2nJ=eK$>ljFQ)k*YN<(>f>>E72fsvso=p1G!)Au8#& z`^;^lw-j_JflKe@m@-{mnbyhPkR!&Iv;2Kd=y$aI>q%ec=XH0TmDjq!rq;iT?x`)c z${bOq{QiKXaEDU*iy^vJXqB8@c5R%!nF0#>3neyipSlR_%1>a|DU~Z{`?gIi$vKXm z)c%MqG68annNnulaUA9z8DxPq_;><%Yg=0w;~UDV~LRNj^XkLv4OUCE5+#%K@ByxUGx62KOakB_6mQDJ4i z_*xpLiPJP~?N`L)DA${!iLpLlfihKNRuKM->3&sq5;BMa3W<1iBra;I(Yp-T6f)P8 z;8Y)}ti}83XuJMg5g8IqE(IaXK8&?xcO&4Ld|zF!+5r|Ae3Gbz=6mZzQu6@MNuJ}l z2xJ<$nW_!eW*<=Wqu0(|rznMBYAIKZLFE&5bK#1pzTI-M)h>L-WpN2qDRMLctU(!H zXVhwH32j?00}^GLv9}3S`f-$L&DA#-ZuoeGx;LC>-syc_tWwv9*Z84r$g990oHwP| z*%U3RGyuZ$v-OJb`Q>d)KPOpL@SmiDv9gP0IU740cGPCOMMd8fjxf2o24%Wxr6N zKfnrIq7OJ+rY5bV|JHJfek>Y)5s{Cq*zVL$4Z?5cb~)pQ1|cGOQnZMvY7YC+hoq!f z)lPAh=7?e>p4jQRnokHmStKCIV9~(7BnZdY`j?@C`B@gC7m2^IxPr6fK0|p41a9Y_ zHQwA8230KB%c9u8%qvm7ut#;KE*iJ#Z*~Psa6ZL5R9P+sg2!Np^ z`%|{VE^O`MjfWWC3rz6m(Wl93ohk`NXyVdM-znIXNDy!=h;+y$XWHn$7xYTRTao)T z=b`%8l?|SLkFwuYXXae zh|Ahx;9Ml|DOsBXkVu;TXWSuQMBNzhrC-^iIUG@5|SEM)f2*u zvsRKkV7We&;Wjsp5l1$;7V+)auoPOvH1|Bex^zeD^_3=C>M%F{AhdwPSfXUYBF2?Z z+E5djVTkUawo4pnzckvG%^6ik4KmyN>O{-NEQ?tp&Slz!UKh`4fB0y(bf;xOM8-Xs znqEevgy5|DUz0R6eq;~$_OFy0$~BbeBwbGU^{Ke^M}JJ2Wn`kwouV%nsZocHBzJg4 zS8`SSA?Xp9mdEa$RFlf*N>u;a{ppW`+_B84**&8pFJ@i&XR`hx71MyH*Gx_$ZE}#S zd-r;Qbmj5P+65YZcYmGmXzH z1z-N?S2?gHP%vbK?Y)g=Er#jXUZ6Z-SxOKK)bW|8@(CLzjrI$sFz~mLGs7h12IpVt zm`*s%pUlp#0(5W*x)D@Ctky2@4Mn^YL`RvjugjF_#lH&?+)k)6nrh+yhy?4f-@2-( zPW2rCe1E(=cQ(-)Pw&GXQDe;W$%RP~Z*%+PAZx2k&TtMdW?D^$gp9^BktEVbJ%}sG zbZLo{VuiRezJyHCPH&b*Ymv~UJ(`DGv?y-rl+~+O*g9()nA>_tUc2s@J>N)8&RCQ!HD<=SGCAS_*Pb!j9$cVh0#d^Mux1$*+{I5 z7DVi7NEBm9W#*S*>2BJuYppQVKO`}Bu3%LMX^UVkzQK5+h`_^8S!4uDxk`s`vjY37 z0#%*>_+~HxgU?D=mj)F2AiL*Z2#U@O;RTbN9VqMwjD3<`_9@FSnd`{mAqGV8_ZO0| z^CTj5wXw@?HU{tSRi?wR3U!~Zms3_NUWD zX524Is$2R8O=x`dHG}4L-~WuhERjLXVIDQ|+kRZNoup*F(I)z@%MZ#{qM42f{Iu~A zE8zm0OF5klH1CPYj&I#a=c!O7Q5*`ana(9KYCwxQq%|s+QWfX+MtF2w-ek%}otiF{YE^K1FLAr%#7rqy=}KDhcsk z1T%KSvIi89P8DUbGTqiGT$7%NeQw>DQdo*F3wfjqMED(TP@@BAw18_tWO26D`*2cY zX~f(=BADMfkyBN7PoHD+f|S#>h$q_%Xl`$<4H>K3iRZWU&aQ8Nx^cPyaZhet0D?p8 zPD_D^%*%wpmoH!5e{#G4`&QFXZSIDR@vYP_nXA9Idu&Y)Rc_haoMoUV+m@$$UzhcR zyJY)UND#pHt!bA8P=AjQe3~dKtj2JNRX&8eM9~Gcs7aKXNH%2|+S9LaqQrtQ8OO-X za*2fnK^X@B&li2dE>>XqE1jAt%eo?tZBcQG-6ISra{Z1dU;-wj04p!QqaM0haK8n- zna}m!>SC_V#7aqIU6itzRhmEP;ViM4Bk_x88!M%_a7T1;kWY`)$aqP11ZE|oM#v$5C5^0BW_4I_L#bKJ*!a@@&}wM2^V(Z0?~DVWY-w}OTKkpBJ= zbfU}z|6x0$ZCGuNzhFc*nDx^ZuipxbMh(}u?=7*+-(as-mw_%uQ~=<#`aCC#&BsJ< zC+-%U`I0eNw@&H8+I9Y5C$W+3y@}{-?{n}ehWI1rW;1Rf9!AOg;vDZI#8;LuF}2x! z5Q{E&65{$wkQ=}@C>HX@H0=rgd`T zwbTd#4n7l&89LY&T=3__lvT*k2UBq{JZasW)Bx8}_RcCyc{|()3WoI(7@`e_H6XXm zE&NlqS#|KuKW;sl>uOW;3$biUe!YuLW~?7h?NWupKQ-npa%Hl%3Ejzp6v5{7DM%>z zvs02JmjbXbJeEW29~8w{hheP?H}{J3V;PZTekLa*cjMMEiMB$~NQ zKKTJn?bN%6!Yo{}M1x>>CFnpY2oY*sPC*?^&@h|-KH)98Lomh`e8-559EnW$Dn~aF zUsB@rn>MNpyKLpNxfMc1f{zpMveM?Y0rhV#g>>cNpcHmd*k)*`9ZVQA=MF_lmEpQB zB+m|nD6-)gQHjHPQ?WbtwIkwmo_Mrg)nAs7NfPHo9ZNbBqg{sf4d1}aDTUWL2~SW9 z3~#_;MQC4ZFJUP5FlHuTfk7?PracN)IWSX~dSdL3AcuEfmfK=_eXhySvwdkb?^CVh z=d6q%6t%`!Q*vR?=OsR>s=R!X=bF<79$EmnVIL%R44F4n-bH4%5!@O#$08>iz?YI1 zc-x@|XWlFZvKPJm{uajFfWWQKKh4Mlc=57nFV^DOjftY#0e169&fz54urfg5`e`69 zC+@%&i40y{T_%k6eRL`DH82m%^G^{z;{GthhfwH0tDp9@N0I*=^?QMxMxTEs7iopP z5k2e)!?n_cC_!#h07lYk4@xWOC9$I0aPi4ej=PU|WoRwuk&ozPQpJANo2!z2MK;cWW;#ogIZa0G4hr?|h71L$A8^1OV96)?^WNT#QZbrv6XgAAI)DVf*5p z{D2I@A7-+`eT!V>b07 zf*GoCx_*;LW^Z;X0Jd|)_9`Kj-w)XqS#cM<%|MqiUXdGO%%<dbWdd72bqsqQF@Xic@l`)|3C8;1q6e}w3h(vN1DujB zVM8q+&nA=W*0&I6d56I|CGGL`U6_qlY)wtsfV4ty7TZD|dvgR_H5u*O=}Crw+eWENqh}W5JWWTrw%y|WVq=HpWiAfW1Wj|W1i*%R5wE~$@;Koa{mwS zMpjhuEkGPqm|dD8uP;5?dGjLyE6GQS(4i}G!&_LskMJ$Cy)a1q z*626>hML>ZtC*BAm}{{9YmF_i4afTGNUk5TqEL;A>$v;hZ`>+Anl!lk@!S>%xy=r| zEe_{4F1o(elk$;00pV`#m5J3A)@Ud7e9%g{;%Z|r2ex_K7)#>AYen{bm)%Ur8r|Es z?#tjNm;+7vhT#LZDRwCC`2y+&R-MRA)WB(cx%Ahq>r=a$_!UdQ9k-v6LS}G4NbaOG z++0J~0fquYf@?p#y*?t0dU+5ELlwl^`oI+m6cm%a3>58ONoB-`lu3H_XMnD$L!WwjZs_uH#k`}5z2_V+pRJa^6pO Date: Tue, 16 Jun 2026 17:34:14 +0530 Subject: [PATCH 07/22] inference: text generation loop sampling --- train_test/test.cpp | 96 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 train_test/test.cpp diff --git a/train_test/test.cpp b/train_test/test.cpp new file mode 100644 index 0000000..41e25e7 --- /dev/null +++ b/train_test/test.cpp @@ -0,0 +1,96 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +bool running = true; +void handle_sigint(int) +{ + std::cout << "\n\n[Stopped by user]\n"; + running = false; +} + +std::map build_vocab(const std::string &path) +{ + std::ifstream file(path); + if (!file.is_open()) + { + std::cerr << "[Error] Cannot open: " << path << "\n"; + exit(1); + } + std::string text((std::istreambuf_iterator(file)), std::istreambuf_iterator()); + + std::set char_set(text.begin(), text.end()); + std::vector chars(char_set.begin(), char_set.end()); + std::sort(chars.begin(), chars.end()); + + std::map it; + for (int i = 0; i < (int)chars.size(); i++) + it[i] = chars[i]; + + return it; +} + +int main() +{ + signal(SIGINT, handle_sigint); + + std::string model_path = "../best_model_script.pt"; + std::string cleaned_path = "../cleaned.txt"; + + std::cout << "--- Loading Pre-trained Model ---\n"; + torch::jit::script::Module model; + try + { + model = torch::jit::load(model_path, torch::kCPU); + } + catch (const c10::Error &e) + { + std::cerr << "[Error] Could not load model: " << e.what() << "\n"; + return 1; + } + model.eval(); + + auto it = build_vocab(cleaned_path); + std::cout << "[INFO] Vocab size : " << it.size() << "\n"; + std::cout << "[INFO] Device : CPU\n"; + std::cout << "Model loaded. Generating text (Ctrl+C to stop)...\n\n"; + std::cout << std::string(50, '-') << "\n"; + + const int block_size = 128; + auto context = torch::zeros({1, 1}, torch::kLong); + + torch::NoGradGuard no_grad; + + while (running) + { + auto idx_cond = context; + if (context.size(1) > block_size) + idx_cond = context.slice(1, context.size(1) - block_size); + + std::vector inputs = {idx_cond}; + auto output = model.forward(inputs).toTuple(); + auto logits = output->elements()[0].toTensor(); + + logits = logits.select(1, logits.size(1) - 1); + auto probs = torch::softmax(logits, -1); + auto idx_next = torch::multinomial(probs, 1); + + context = torch::cat({context, idx_next}, 1); + if (context.size(1) > block_size) + context = context.slice(1, context.size(1) - block_size); + + int token = idx_next[0][0].item(); + if (it.count(token)) + std::cout << it[token] << std::flush; + else + std::cout << '?' << std::flush; + } + + return 0; +} \ No newline at end of file From 624d4af1fa24d67272205a446556312e231d311b Mon Sep 17 00:00:00 2001 From: Eamon Date: Tue, 16 Jun 2026 17:39:14 +0530 Subject: [PATCH 08/22] model architecture and sync tokenizer with main to export weights --- train_test/model.py | 131 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 train_test/model.py diff --git a/train_test/model.py b/train_test/model.py new file mode 100644 index 0000000..33425e9 --- /dev/null +++ b/train_test/model.py @@ -0,0 +1,131 @@ +import torch +import torch.nn as nn +from torch.nn import functional as F +from typing import Optional, Tuple +import os +import tiktoken + +# File Paths +_project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_model_path = os.path.join(_project_root, 'best_model.pt') +_script_path = os.path.join(_project_root, 'best_model_script.pt') + +# Hyperparameters (Synced perfectly with main.py) +block_size = 12 +n_embd = 64 +n_head = 4 +n_layer = 4 +dropout = 0.1 +device = 'cuda' if torch.cuda.is_available() else 'cpu' + +# Tokenizer (Updated to use Tiktoken GPT-2 encoding from enginw/main.py) +tokenizer = tiktoken.get_encoding("gpt2") +vocab_size = tokenizer.n_vocab +def encode(s): return tokenizer.encode(s) +def decode(l): return tokenizer.decode(l) + + +chars = [] + + +# Model classes +class Head(nn.Module): + def __init__(self, head_size: int): + super().__init__() + self.key = nn.Linear(n_embd, head_size, bias=False) + self.query = nn.Linear(n_embd, head_size, bias=False) + self.value = nn.Linear(n_embd, head_size, bias=False) + self.register_buffer('tril', torch.tril( + torch.ones(block_size, block_size))) + self.dropout = nn.Dropout(dropout) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + B, T, C = x.shape + k = self.key(x) + q = self.query(x) + wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5 + wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf')) + wei = F.softmax(wei, dim=-1) + wei = self.dropout(wei) + return wei @ self.value(x) + + +class MultiHeadAttention(nn.Module): + def __init__(self, num_heads: int, head_size: int): + super().__init__() + self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)]) + self.proj = nn.Linear(head_size * num_heads, n_embd) + self.dropout = nn.Dropout(dropout) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + out = torch.cat([h(x) for h in self.heads], dim=-1) + return self.dropout(self.proj(out)) + + +class FeedFoward(nn.Module): + def __init__(self, n_embd: int): + super().__init__() + self.net = nn.Sequential( + nn.Linear(n_embd, 4 * n_embd), + nn.ReLU(), + nn.Linear(4 * n_embd, n_embd), + nn.Dropout(dropout), + ) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.net(x) + + +class Block(nn.Module): + def __init__(self, n_embd: int, n_head: int): + super().__init__() + head_size = n_embd // n_head + self.sa = MultiHeadAttention(n_head, head_size) + self.ffwd = FeedFoward(n_embd) + self.ln1 = nn.LayerNorm(n_embd) + self.ln2 = nn.LayerNorm(n_embd) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + self.sa(self.ln1(x)) + x = x + self.ffwd(self.ln2(x)) + return x + + +class GPTLanguageModel(nn.Module): + def __init__(self): + super().__init__() + self.token_embedding_table = nn.Embedding(vocab_size, n_embd) + self.position_embedding_table = nn.Embedding(block_size, n_embd) + self.blocks = nn.Sequential( + *[Block(n_embd, n_head=n_head) for _ in range(n_layer)]) + self.ln_f = nn.LayerNorm(n_embd) + self.lm_head = nn.Linear(n_embd, vocab_size) + self.apply(self._init_weights) + + def _init_weights(self, module: nn.Module) -> None: + if isinstance(module, nn.Linear): + torch.nn.init.normal_(module.weight, mean=0.0, std=0.02) + if module.bias is not None: + torch.nn.init.zeros_(module.bias) + elif isinstance(module, nn.Embedding): + torch.nn.init.normal_(module.weight, mean=0.0, std=0.02) + + def forward(self, idx: torch.Tensor, targets: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, Optional[torch.Tensor]]: + B, T = idx.shape + tok_emb = self.token_embedding_table(idx) + pos_emb = self.position_embedding_table( + torch.arange(T, device=idx.device)) + x = tok_emb + pos_emb + x = self.blocks(x) + x = self.ln_f(x) + logits = self.lm_head(x) + + if targets is None: + loss = None + else: + B, T, C = logits.shape + logits = logits.view(B * T, C) + targets = targets.view(B * T) + loss = F.cross_entropy(logits, targets) + + return logits, loss From b82c47bc15e51e461c91d36ce0202b23ebe7b266 Mon Sep 17 00:00:00 2001 From: Eamon Sippy Date: Tue, 16 Jun 2026 17:38:58 +0530 Subject: [PATCH 09/22] Add CI approval workflow configuration --- .github/workflows/{ci.yml => ci-approval.yml} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename .github/workflows/{ci.yml => ci-approval.yml} (99%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci-approval.yml similarity index 99% rename from .github/workflows/ci.yml rename to .github/workflows/ci-approval.yml index 992ef59..c4d6542 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci-approval.yml @@ -1,4 +1,4 @@ -name: CI +name: ci-approval on: workflow_dispatch: From 0835a76ba56d485cedf3321b8f56510b6b637b23 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 16 Jun 2026 17:45:30 +0530 Subject: [PATCH 10/22] chore(deps): bump hadolint/hadolint-action from 3.1.0 to 3.3.0 (#70) Bumps [hadolint/hadolint-action](https://github.com/hadolint/hadolint-action) from 3.1.0 to 3.3.0. - [Release notes](https://github.com/hadolint/hadolint-action/releases) - [Commits](https://github.com/hadolint/hadolint-action/compare/v3.1.0...v3.3.0) --- updated-dependencies: - dependency-name: hadolint/hadolint-action dependency-version: 3.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/pr-check.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/pr-check.yml b/.github/workflows/pr-check.yml index b50de6b..944dfe2 100644 --- a/.github/workflows/pr-check.yml +++ b/.github/workflows/pr-check.yml @@ -199,19 +199,19 @@ jobs: exit $failed - name: Lint — Dockerfile.cpp - uses: hadolint/hadolint-action@v3.1.0 + uses: hadolint/hadolint-action@v3.3.0 with: dockerfile: .devops/Dockerfile.cpp failure-threshold: error - name: Lint — Dockerfile (CPU) - uses: hadolint/hadolint-action@v3.1.0 + uses: hadolint/hadolint-action@v3.3.0 with: dockerfile: .devops/Dockerfile failure-threshold: error - name: Lint — Dockerfile.backend (CUDA) - uses: hadolint/hadolint-action@v3.1.0 + uses: hadolint/hadolint-action@v3.3.0 with: dockerfile: .devops/Dockerfile.backend failure-threshold: error From 5dffd41ef2d52ccc67aa48656f030fdee25ec234 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 17 Jun 2026 06:44:04 +0000 Subject: [PATCH 11/22] chore(deps): bump pydantic from 2.10.4 to 2.13.4 in /backend Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.10.4 to 2.13.4. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](https://github.com/pydantic/pydantic/compare/v2.10.4...v2.13.4) --- updated-dependencies: - dependency-name: pydantic dependency-version: 2.13.4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- backend/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/requirements.txt b/backend/requirements.txt index cec57cc..9275f43 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -1,6 +1,6 @@ fastapi==0.115.6 uvicorn[standard]==0.34.0 -pydantic==2.10.4 +pydantic==2.13.4 pydantic-settings==2.7.1 httpx==0.28.1 redis==5.2.1 From 7c580789d2b2ad0d53035b438fe6b946872226c7 Mon Sep 17 00:00:00 2001 From: Eamon Date: Thu, 18 Jun 2026 16:04:16 +0530 Subject: [PATCH 12/22] Mark frontend vendored and add pytorch submodule Update .gitattributes to mark frontend/** as linguist-vendored and add a submodule stanza for PyTorch (path: pytorch, url: https://github.com/pytorch/pytorch). This records the PyTorch submodule location and ensures frontend files are treated as vendored by GitHub linguist. --- .gitattributes | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.gitattributes b/.gitattributes index 251a617..5f080b8 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1 +1,6 @@ frontend/** linguist-vendored +frontend/** linguist-vendored +[submodule "pytorch"] + path = pytorch + url = https://github.com/pytorch/pytorch + From b5b86c1f24926c1c5d1f9c125489f3f569585e1f Mon Sep 17 00:00:00 2001 From: Eamon Date: Thu, 18 Jun 2026 16:08:10 +0530 Subject: [PATCH 13/22] Add pytoch/pytorch file --- pytorch/pytorch | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 pytorch/pytorch diff --git a/pytorch/pytorch b/pytorch/pytorch new file mode 100644 index 0000000..e69de29 From eb41598a7a5ec6c0666e7a677c958a7c834b0693 Mon Sep 17 00:00:00 2001 From: Eamon Sippy Date: Thu, 18 Jun 2026 16:06:58 +0530 Subject: [PATCH 14/22] Delete libtorch directory --- pytorch/pytorch | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 pytorch/pytorch diff --git a/pytorch/pytorch b/pytorch/pytorch deleted file mode 100644 index e69de29..0000000 From c909f47bf1b3476408c9498e7a51c26d135db119 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 20 Jun 2026 00:45:16 +0530 Subject: [PATCH 15/22] chore(deps): bump docker/metadata-action from 5 to 6 (#55) Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5 to 6. - [Release notes](https://github.com/docker/metadata-action/releases) - [Commits](https://github.com/docker/metadata-action/compare/v5...v6) --- updated-dependencies: - dependency-name: docker/metadata-action dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eamon Sippy --- .github/workflows/ci-approval.yml | 6 +++--- .github/workflows/docker-publish.yml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci-approval.yml b/.github/workflows/ci-approval.yml index c4d6542..1eae787 100644 --- a/.github/workflows/ci-approval.yml +++ b/.github/workflows/ci-approval.yml @@ -143,7 +143,7 @@ jobs: - name: Extract metadata id: meta - uses: docker/metadata-action@v5 + uses: docker/metadata-action@v6 with: images: ${{ env.IMAGE_PREFIX }}-cpp tags: | @@ -190,7 +190,7 @@ jobs: - name: Extract metadata id: meta - uses: docker/metadata-action@v5 + uses: docker/metadata-action@v6 with: images: ${{ env.IMAGE_PREFIX }}-cpu tags: | @@ -236,7 +236,7 @@ jobs: - name: Extract metadata id: meta - uses: docker/metadata-action@v5 + uses: docker/metadata-action@v6 with: images: ${{ env.IMAGE_PREFIX }}-cuda tags: | diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml index 0986534..ca33ce7 100644 --- a/.github/workflows/docker-publish.yml +++ b/.github/workflows/docker-publish.yml @@ -57,7 +57,7 @@ jobs: - name: Extract metadata id: meta - uses: docker/metadata-action@v5 + uses: docker/metadata-action@v6 with: images: ${{ env.IMAGE_PREFIX }}-cpp tags: | @@ -104,7 +104,7 @@ jobs: - name: Extract metadata id: meta - uses: docker/metadata-action@v5 + uses: docker/metadata-action@v6 with: images: ${{ env.IMAGE_PREFIX }}-cpu tags: | @@ -150,7 +150,7 @@ jobs: - name: Extract metadata id: meta - uses: docker/metadata-action@v5 + uses: docker/metadata-action@v6 with: images: ${{ env.IMAGE_PREFIX }}-cuda tags: | From d9be5b1c7aba0697bb4d24728a866304dc754602 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 19 Jun 2026 19:16:54 +0000 Subject: [PATCH 16/22] chore(deps): bump docker/login-action from 3 to 4 Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/v3...v4) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/ci-approval.yml | 6 +++--- .github/workflows/docker-publish.yml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci-approval.yml b/.github/workflows/ci-approval.yml index 1eae787..84b9e6b 100644 --- a/.github/workflows/ci-approval.yml +++ b/.github/workflows/ci-approval.yml @@ -135,7 +135,7 @@ jobs: - name: Login to GHCR if: ${{ inputs.push_image == 'true' }} - uses: docker/login-action@v3 + uses: docker/login-action@v4 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} @@ -182,7 +182,7 @@ jobs: - name: Login to GHCR if: ${{ inputs.push_image == 'true' }} - uses: docker/login-action@v3 + uses: docker/login-action@v4 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} @@ -228,7 +228,7 @@ jobs: - name: Login to GHCR if: ${{ inputs.push_image == 'true' }} - uses: docker/login-action@v3 + uses: docker/login-action@v4 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml index ca33ce7..7fc9225 100644 --- a/.github/workflows/docker-publish.yml +++ b/.github/workflows/docker-publish.yml @@ -49,7 +49,7 @@ jobs: - name: Login to GHCR if: ${{ inputs.push_image == 'true' }} - uses: docker/login-action@v3 + uses: docker/login-action@v4 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} @@ -96,7 +96,7 @@ jobs: - name: Login to GHCR if: ${{ inputs.push_image == 'true' }} - uses: docker/login-action@v3 + uses: docker/login-action@v4 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} @@ -142,7 +142,7 @@ jobs: - name: Login to GHCR if: ${{ inputs.push_image == 'true' }} - uses: docker/login-action@v3 + uses: docker/login-action@v4 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} From c328d5f37edc6aa58e5cd58ec282ef7564eec093 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 19 Jun 2026 19:21:47 +0000 Subject: [PATCH 17/22] chore(deps): bump docker/setup-qemu-action from 3 to 4 Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3 to 4. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v3...v4) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/ci-approval.yml | 4 ++-- .github/workflows/docker-publish.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci-approval.yml b/.github/workflows/ci-approval.yml index 84b9e6b..a50a9c1 100644 --- a/.github/workflows/ci-approval.yml +++ b/.github/workflows/ci-approval.yml @@ -127,7 +127,7 @@ jobs: steps: - uses: actions/checkout@v4 - - uses: docker/setup-qemu-action@v3 + - uses: docker/setup-qemu-action@v4 - uses: docker/setup-buildx-action@v3 - name: Set lowercase image prefix @@ -174,7 +174,7 @@ jobs: steps: - uses: actions/checkout@v4 - - uses: docker/setup-qemu-action@v3 + - uses: docker/setup-qemu-action@v4 - uses: docker/setup-buildx-action@v3 - name: Set lowercase image prefix diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml index 7fc9225..ad72018 100644 --- a/.github/workflows/docker-publish.yml +++ b/.github/workflows/docker-publish.yml @@ -41,7 +41,7 @@ jobs: steps: - uses: actions/checkout@v4 - - uses: docker/setup-qemu-action@v3 + - uses: docker/setup-qemu-action@v4 - uses: docker/setup-buildx-action@v3 - name: Set lowercase image prefix @@ -88,7 +88,7 @@ jobs: steps: - uses: actions/checkout@v4 - - uses: docker/setup-qemu-action@v3 + - uses: docker/setup-qemu-action@v4 - uses: docker/setup-buildx-action@v3 - name: Set lowercase image prefix From b2b4640654827376f336c8c365d866f31a4a03e1 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 20 Jun 2026 13:49:26 +0000 Subject: [PATCH 18/22] chore(deps): bump docker/setup-buildx-action from 3 to 4 Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/ci-approval.yml | 6 +++--- .github/workflows/docker-publish.yml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci-approval.yml b/.github/workflows/ci-approval.yml index a50a9c1..aaf3b52 100644 --- a/.github/workflows/ci-approval.yml +++ b/.github/workflows/ci-approval.yml @@ -128,7 +128,7 @@ jobs: - uses: actions/checkout@v4 - uses: docker/setup-qemu-action@v4 - - uses: docker/setup-buildx-action@v3 + - uses: docker/setup-buildx-action@v4 - name: Set lowercase image prefix run: echo "IMAGE_PREFIX=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/quadtrix" >> $GITHUB_ENV @@ -175,7 +175,7 @@ jobs: - uses: actions/checkout@v4 - uses: docker/setup-qemu-action@v4 - - uses: docker/setup-buildx-action@v3 + - uses: docker/setup-buildx-action@v4 - name: Set lowercase image prefix run: echo "IMAGE_PREFIX=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/quadtrix" >> $GITHUB_ENV @@ -221,7 +221,7 @@ jobs: steps: - uses: actions/checkout@v4 - - uses: docker/setup-buildx-action@v3 + - uses: docker/setup-buildx-action@v4 - name: Set lowercase image prefix run: echo "IMAGE_PREFIX=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/quadtrix" >> $GITHUB_ENV diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml index ad72018..205c599 100644 --- a/.github/workflows/docker-publish.yml +++ b/.github/workflows/docker-publish.yml @@ -42,7 +42,7 @@ jobs: - uses: actions/checkout@v4 - uses: docker/setup-qemu-action@v4 - - uses: docker/setup-buildx-action@v3 + - uses: docker/setup-buildx-action@v4 - name: Set lowercase image prefix run: echo "IMAGE_PREFIX=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/quadtrix" >> "$GITHUB_ENV" @@ -89,7 +89,7 @@ jobs: - uses: actions/checkout@v4 - uses: docker/setup-qemu-action@v4 - - uses: docker/setup-buildx-action@v3 + - uses: docker/setup-buildx-action@v4 - name: Set lowercase image prefix run: echo "IMAGE_PREFIX=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/quadtrix" >> "$GITHUB_ENV" @@ -135,7 +135,7 @@ jobs: steps: - uses: actions/checkout@v4 - - uses: docker/setup-buildx-action@v3 + - uses: docker/setup-buildx-action@v4 - name: Set lowercase image prefix run: echo "IMAGE_PREFIX=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/quadtrix" >> "$GITHUB_ENV" From 4b2fb2c6ab593b63620dad689cf0883a39a8d049 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 20 Jun 2026 13:49:26 +0000 Subject: [PATCH 19/22] chore(deps): bump actions/upload-artifact from 4 to 7 Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v7) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/ci-approval.yml | 4 ++-- .github/workflows/pr-check.yml | 2 +- .github/workflows/release.yml | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci-approval.yml b/.github/workflows/ci-approval.yml index a50a9c1..c2ccda6 100644 --- a/.github/workflows/ci-approval.yml +++ b/.github/workflows/ci-approval.yml @@ -84,7 +84,7 @@ jobs: run: ./quadtrix --help || true - name: Upload binary - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: quadtrix-linux-amd64 path: quadtrix @@ -111,7 +111,7 @@ jobs: run: tar -czf quadtrix-macos-arm64.tar.gz quadtrix - name: Upload binary - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: quadtrix-macos-arm64 path: quadtrix-macos-arm64.tar.gz diff --git a/.github/workflows/pr-check.yml b/.github/workflows/pr-check.yml index 944dfe2..07b05f2 100644 --- a/.github/workflows/pr-check.yml +++ b/.github/workflows/pr-check.yml @@ -150,7 +150,7 @@ jobs: run: ./quadtrix --help || true - name: Upload artifact - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: ${{ matrix.artifact }} path: quadtrix diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index e92b486..05fd17b 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -93,7 +93,7 @@ jobs: tar -czf "${package}.tar.gz" -C "${ARTIFACT_ROOT}" "${package}" - name: Upload artifacts - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: quadtrix-bin-ubuntu-${{ matrix.build }}-cpu path: quadtrix-${{ needs.release-metadata.outputs.tag_name }}-bin-ubuntu-${{ matrix.build }}-cpu.tar.gz @@ -142,7 +142,7 @@ jobs: Compress-Archive -Path "${env:ARTIFACT_ROOT}\${package}\*" -DestinationPath "${package}.zip" -Force - name: Upload artifacts - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: quadtrix-bin-windows-${{ matrix.arch }}-cpu path: quadtrix-${{ needs.release-metadata.outputs.tag_name }}-bin-windows-${{ matrix.arch }}-cpu.zip @@ -192,7 +192,7 @@ jobs: tar -czf "${package}.tar.gz" -C "${ARTIFACT_ROOT}" "${package}" - name: Upload artifacts - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: quadtrix-bin-macos-${{ matrix.build }}-cpu path: quadtrix-${{ needs.release-metadata.outputs.tag_name }}-bin-macos-${{ matrix.build }}-cpu.tar.gz From 35b58e56a69efef5a7aa2eba5a94eb0a17907680 Mon Sep 17 00:00:00 2001 From: Eamon Sippy Date: Fri, 26 Jun 2026 12:42:53 +0530 Subject: [PATCH 20/22] Revise README for improved clarity and details Updated README to clarify project structure and usage instructions. --- README.md | 208 +++++++++++++++++++++++++++--------------------------- 1 file changed, 104 insertions(+), 104 deletions(-) diff --git a/README.md b/README.md index 59709db..dda598e 100644 --- a/README.md +++ b/README.md @@ -1,140 +1,140 @@ # Quadtrix.cpp -

-image +Language models in simple, dependency-free C++, with no need for 245MB of PyTorch or 107MB of cPython to understand how a transformer actually works. The native path is a from-scratch decoder-only GPT: tensors, embeddings, multi-head causal self-attention, layer norm, cross-entropy, and a analytical backward pass with AdamW, all in [main.cpp](main.cpp) and [include/](include/). No autograd, no framework — every gradient is derived and written out. +***technical notes***: [docs](https://eamon2009.github.io/LLMs/) +Alongside it sits a parallel PyTorch implementation in [engine/main.py](engine/main.py) and [engine/inference.py](engine/inference.py), so you can train and generate the same architecture with `torch` + `tiktoken` when you want speed instead of transparency. A FastAPI middleware layer in [backend/](backend/) and a React/TypeScript web UI in [frontend/](frontend/) let you chat with either backend in the browser. There's also an experimental integrated-GPU path in [iGPU/](engine/iGPU/). -image -

+The point of this repo is the C++ core. The PyTorch, FastAPI, and frontend layers exist to make the model usable, but if you're here to learn how a GPT is actually built and trained without a framework doing the work for you, [include/backward.h](include/backward.h) is where to start reading. -Quadtrix.cpp is an experimental local LLM project that combines training, inference, and a chat interface in a single repository across three execution paths: a C++ versions, a PyTorch backend, and a React frontend.The C++ version is a decoder-only transformer - no external tensor libraries, no automatic differentiation. It includes character-level tokenization, backpropagation, AdamW optimization, checkpointing, and autoregressive generationd. The PyTorch version offers faster training and inference with BPE tokenization via tiktoken, while the React frontend provides a web UI on top of python backend.You can train small character-level models on CPU or Colab, or move to BPE-based models with the PyTorch backend. All version are designed to be hackable and runnable on consumer hardware without requiring massive GPU clusters. +## quick start (C++, train + chat) -***technical notes***: [docs](https://eamon2009.github.io/LLMs/) +The fastest way to see the whole pipeline — tokenize, train, checkpoint, generate — using the bundled character-level corpus: -## Leaderboard +```bash +g++ -std=c++17 -O2 -I. -Iinclude -o quadtrix.exe main.cpp +./quadtrix.exe data/input.txt +``` -Runs are ranked by validation loss. Lower is better. +This trains from scratch on `data/input.txt` and writes the best checkpoint to `best_model.bin`. Once you have a checkpoint, generate or chat with it: -| # | Val Loss | Parameters | Time | Hardware | Description | -|--:|----------|------------|----------|----------|------------------------------------------| -| 1 | **0.7176** | 10.82M | 61.3 min | NVIDIA T4 | Large-scale run, coherent paragraphs, strong convergence | -| 2 | 0.9250 | 1.99M | 6.1 min | NVIDIA T4 | Optimised run, fast training, stable learning | -| 3 | 1.3145 | 0.82M | 39.4 min | x64 CPU | Baseline, small data | -| 4 | 1.6371 | 0.82M | 76.2 min | x64 CPU | Extended CPU training, 3000 iterations | +```bash +./quadtrix.exe data/input.txt --generate +./quadtrix.exe data/input.txt --chat --chat-tokens 300 +``` -All runs: @Eamon2009, 2026. +debugging tip: drop `-O2` for `-g` when compiling if you want to step through `include/backward.h` or `include/gpt.h` in a debugger — the manual backward pass is much easier to follow one breakpoint at a time. -## Benchmarks +### runtime arguments -### Runs at a Glance +```bash +quadtrix.exe [data_path] [--generate] [--chat] [--chat-tokens N] +``` -| Metric | Character-Level | Small Scale | Large Scale | -|-------------------|------------------|-------------|-------------| -| Parameters | 0.83M | 2.00M | 19.17M | -| Layers | 4 | 4 | 4 | -| Embedding dim | 128 | 200 | 200 | -| Attention heads | 4 | 4 | 4 | -| Context length | 64 | 200 | 200 | -| Vocab | 105 char | 110 char | ~50K BPE | -| Corpus | TinyStories | TinyStories | Children's Stories | -| Iterations | 3,000 | 5,000 | 5,000 | -| Train loss | 1.5632 | 0.9045 | — | -| Val loss | 1.6371 | 0.9301 | — | -| Gen. gap | 0.0739 | 0.0256 | — | +| Argument | Description | +|---|---| +| `data_path` | Plain-text corpus used to build the tokenizer and train/validation split | +| `--generate` | Load weights and continuously generate text | +| `--chat` | Load weights and start interactive terminal chat | +| `--chat-tokens N` | Max generated tokens per chat response | + +| Env var | Default | Description | +|---|---|---| +| `GPT_DATA_PATH` | `data/input.txt` | Override the default training corpus | +| `GPT_MODEL_PATH` | `best_model.bin` | Override the checkpoint path | + +## what's actually implemented in C++ + +No third-party runtime dependency — it builds from `main.cpp`, `config/config.h`, and `include/*.h` alone. + +- Character-level tokenizer built directly from the input corpus +- Train/validation split via `DataLoader` +- Token + positional embeddings +- Multi-head causal self-attention with explicit QKV projections +- Pre-layer-norm residual transformer blocks +- Feed-forward MLP with ReLU +- Cross-entropy loss +- **Fully analytical backward pass** — every gradient (attention, layer norm, MLP, embeddings) is derived and coded in `include/backward.h`, not autograd +- AdamW optimizer (first/second moment estimates, weight decay) +- Checkpoint save/load +- Autoregressive generation and terminal chat mode + +Hyperparameters live in `config/config.h` and require a rebuild to take effect: + +```cpp +static const int BATCH_SIZE = 4; +static const int BLOCK_SIZE = 64; +static const int N_EMBD = 128; +static const int N_HEAD = 4; +static const int N_LAYER = 4; +static const float DROPOUT = 0.2f; +static const float LEARNING_RATE = 3e-4f; +static const int MAX_ITERS = 3000; +``` -### Comparison to Related Projects +For an optimized native build: -| Project | Focus | Language | Autograd | -|---------------|--------------------------------------------|-------------------|----------------------| -| nanoGPT | Minimal GPT training | Python | PyTorch | -| minGPT | Educational GPT | Python | PyTorch | -| llama2.c | Inference-oriented C | C | None | -| **Quadtrix.cpp** | Training + inference + web UI + multi-backend | C++ / Python / TypeScript | Manual C++ + PyTorch | +```bash +g++ -std=c++17 -O3 -march=native -I. -Iinclude -o quadtrix.exe main.cpp +``` +## the PyTorch reference path -# Getting started -**Create a Python virtual environment** +[engine/main.py](engine/main.py) trains the same architectural idea with `torch`, `torch.nn`, and GPT-2 BPE tokenization via `tiktoken`, useful when you want to scale past what C++ loops can comfortably train on CPU. -```powershell -python -m venv .venv -.\.venv\Scripts\python.exe -m pip install --upgrade pip +```bash +python engine/main.py ``` -** Install backend and PyTorch dependencies** +It looks for `engine/input.txt` by default; point it elsewhere with `QUADTRIX_TRAIN_DATA` if needed. Run inference against a saved checkpoint: -```powershell -cd backend -..\.venv\Scripts\python.exe -m pip install -r requirements.txt -cd .. -``` -**Train** -```powershell -cd C:\Users\Admin\Documents\GitHub\Quadtrix.cpp -.\.venv\Scripts\python.exe engine\main.py +```bash +python engine/inference.py --checkpoint engine/best_model.pt --prompt "Once upon a time" --max-new-tokens 100 ``` -**Build the C++ executable** -Skip if `quadtrix.exe` already exists. To rebuild: +## web chat (FastAPI + React) -```powershell -g++ -std=c++17 -O2 -I. -Iinclude -o quadtrix.exe main.cpp -``` +To chat with either backend from a browser instead of the terminal, bring up the API and the frontend in two terminals: -For maximum CPU throughput: +```bash +# terminal 1 — backend +cd backend && uvicorn main:app --host 127.0.0.1 --port 3001 -```powershell -g++ -std=c++17 -O3 -march=native -I. -Iinclude -o quadtrix.exe main.cpp +# terminal 2 — frontend +cd frontend && npm run dev ``` ---- +Then open `http://localhost:5173` and select a backend. The PyTorch path works out of the box once a `.pt` checkpoint exists; the C++ backend option expects a compatible HTTP service at `CPP_SERVER_URL` exposing `/health` and `/generate`, which `main.cpp` does not currently serve on its own — use the PyTorch backend for the web UI unless you've built that bridge. +## results so far -## Technical Reference -***Quadtrix is a decoder-only transformer. The architecture follows the standard GPT design with pre-layer normalization, causal self-attention, and residual connections.*** -The C++ version is : +image -- Character-level tokenizer -- Manual tensor operations -- Analytical backpropagation -- AdamW optimizer with bias correction -- Checkpoint save/load -- Autoregressive generation +| Run | Params | Val loss | Time | Notes | +|---|---|---|---|---| +| C++ CPU baseline | 0.82M | 1.31 | 39.4 min | small data, fragmented output | +| C++ CPU extended | 0.83M | 1.64 | 76.2 min | 3,000 iters, char-level, 28.3M train tokens | +| T4 | 10.82M | 0.72 | 61.3 min | coherent paragraphs, strong convergence | +| T4 optimized | 1.99M | 0.93 | 6.1 min | fast, stable, basic coherence | -**The PyTorch path uses torch.nn and tiktoken for faster experimentation and GPU acceleration.** +See [run.md](run.md) and the leaderboard in the full docs for more configurations. -## File Structure -``` -Quadtrix.cpp/ -├── main.cpp -├── config/ -│ └── config.h -├── include/ -│ ├── tensor.h -│ ├── gpt.h -│ └── backward.h -│ └── ... -│ -├── data/ -│ └── input.txt -├── engine/ -│ ├── main.py -│ ├── inference.py -│ └── best_model.pt -├── backend/ -│ ├── main.py -│ ├── inference.py -│ └── requirements.txt -├── frontend/ -│ ├── src/ -│ └── ... -└── quadtrix.exe (after build) -``` +## how this differs from similar projects + +| Project | Focus | Language | Autograd | +|---|---|---|---| +| nanoGPT / minGPT | Minimal, educational GPT training | Python | PyTorch | +| llama2.c | Inference-only | C | None | +| **Quadtrix.cpp** | Training *and* inference, manual backward pass, web UI | C++ / Python / TypeScript | Manual (C++) + PyTorch | + +I'd like the C++ core (`main.cpp`, `include/`, `config/`) to stay dependency-free and to stay the part of this repo that explin transformer internals directly. The PyTorch engine, FastAPI middleware, and React frontend are welcome to grow more features, integrations, and UI polish. If you build a port to another language or framework, I'm happy to link to it from a notable-forks section; just open an issue or PR. + +## references + +- Vaswani et al., "Attention Is All You Need", 2017 +- Radford et al., GPT-2 technical work, 2019 +- nanoGPT and minGPT as educational reference points -## References -- Vaswani et al., "Attention Is All You Need", NeurIPS 2017. -- Radford et al., "Language Models are Unsupervised Multitask Learners", 2019. -- Karpathy, nanoGPT -- Karpathy, minGPT +## license -## License MIT From eb419c81124a057d2a3d5fc4117272b38b2dadd6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 28 Jun 2026 22:55:05 +0530 Subject: [PATCH 21/22] chore(deps): bump redis from 5.2.1 to 8.0.1 in /backend (#93) Bumps [redis](https://github.com/redis/redis-py) from 5.2.1 to 8.0.1. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](https://github.com/redis/redis-py/compare/v5.2.1...v8.0.1) --- updated-dependencies: - dependency-name: redis dependency-version: 8.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- backend/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/requirements.txt b/backend/requirements.txt index 9275f43..960d38f 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -3,6 +3,6 @@ uvicorn[standard]==0.34.0 pydantic==2.13.4 pydantic-settings==2.7.1 httpx==0.28.1 -redis==5.2.1 +redis==8.0.1 torch tiktoken From bfc869aa23b114afac43b06bab414843a593f459 Mon Sep 17 00:00:00 2001 From: Eamon Sippy Date: Tue, 30 Jun 2026 13:15:59 +0530 Subject: [PATCH 22/22] Fix grammar in README.md project description Corrected grammar and phrasing in the project description. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index dda598e..bc88dfd 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Quadtrix.cpp -Language models in simple, dependency-free C++, with no need for 245MB of PyTorch or 107MB of cPython to understand how a transformer actually works. The native path is a from-scratch decoder-only GPT: tensors, embeddings, multi-head causal self-attention, layer norm, cross-entropy, and a analytical backward pass with AdamW, all in [main.cpp](main.cpp) and [include/](include/). No autograd, no framework — every gradient is derived and written out. +Language models in dependency-free C++, with no need for 245MB of PyTorch or 107MB of cPython to make a transformer actually work. The native path is a from-scratch decoder-only GPT: tensors, embeddings, multi-head causal self-attention, layer norm, cross-entropy, and a analytical backward pass with AdamW, all in [main.cpp](main.cpp) and [include/](include/). No autograd, no framework — every gradient is derived and written out. ***technical notes***: [docs](https://eamon2009.github.io/LLMs/) Alongside it sits a parallel PyTorch implementation in [engine/main.py](engine/main.py) and [engine/inference.py](engine/inference.py), so you can train and generate the same architecture with `torch` + `tiktoken` when you want speed instead of transparency. A FastAPI middleware layer in [backend/](backend/) and a React/TypeScript web UI in [frontend/](frontend/) let you chat with either backend in the browser. There's also an experimental integrated-GPU path in [iGPU/](engine/iGPU/).