Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Discussion: Cortex.cpp Model and model.yaml #1090

Closed

Description

@dan-menlo

Overview

Docs

Activity

  1. converted this from a draft issue on Sep 4, 2024
  2. freelerobot commented on Sep 4, 2024

    @freelerobot
    Contributor

    Data Folder Questions

    1. What is the data structure of ~/.cortexcpp/models?
      Where do the following go
    • model yaml
    • model binaries, especially of multiple binaries (model_1_of_5.bin)
    • versions of the same model, e.g. llama3.1, llama3.2
    • presets (if any, can also defer for later discussion)
    1. Is our preference on a more flat folder structure? rather than super nested?

    2. Previously, we had many bugs resulting from expecting folder & file names to be a certain way.
      e.g. we expected unique_model_id (used by backend) to be the same as the model folder name
      Something to be aware of in this iteration 🙏

    Model Downloading

    1. What happens when model download fails halfway (e.g. internet disconnected)?
    2. How do we detect models? e.g. if users "import models locally" would it still work
    3. How do we version models?
    • If we update our model.yaml, or remote model binary in the HF branch, will download still work?
    • Or will "redownloading/updating" fail due to "model exists"
    1. Letting users do cortex models update is currently out of scope right?

    Model importing

    1. Can users import existing models?
    2. Is it hard copy or symlink

    Model YAML

    1. We are auto populating the model.yaml if user downloads a new GGUF file (not from our HF repo)?
    2. What happens when user deletes the YAML accidentally?
    3. What happens when YAML is there but Binary is deleted? (should we spec little unit tests like this)
    4. Is this up to date? https://cortex.so/docs/model-yaml/
    5. If users update YAMLs, when do changes take effect?
  3. vansangpfiev commented on Sep 5, 2024

    @vansangpfiev
    Contributor

    Data Folder Questions

    Q 1. What is the data structure of ~/.cortexcpp/models?

    ~/.cortexcpp/
    |___ models
           |__ tinyllama.yaml
           |__ tinyllama
           |     |__ model_01.gguf
           |     |__ model_02.gguf
           |     |__ model.yml
           |__ llama3.1
           |__ llama3.2
    

    After downloading model (from cortexso or other HF repository), cortex generates tinyllama.yaml file which is used for model management.

    Q 2. Is our preference on a more flat folder structure? rather than super nested?
    Can you give an example of the flat folder structure?
    Q 3. Can you elaborate more about the issue? Do we have any ticket to track that issue yet?

    Model importing
    Q 1. Can users import existing models?

    • We don't support it yet. TBD when we will support it.
      Q 2. Is it hard copy or symlink
    • It will be a symlink

    cc: @0xSage

  4. nguyenhoangthuan99 commented on Sep 5, 2024

    @nguyenhoangthuan99
    Contributor

    Model Yaml

    1. Folder structure:
    ~/.cortexcpp/
    |___ models
           |__ tinyllama.yaml
           |__ tinyllama
           |     |__ model_01.gguf
           |     |__ model_02.gguf
           |     |__ model.yml
           |__ llama3.1
           |__ llama3.2
    
    1. With model not from cortexso:
    • When model is downloaded, it will parse and save information to <model_id>/model.yml and <model_id>.yaml, this 2 files is the same.
    • When user delete 1 of 2 .yaml file. We can provide a command to recover it like cortex-cpp models recover, to check and resolve yml error.
    1. when YAML is there but binary is deleted, will raise No such file or directory error when load models.
    2. The doc from https://cortex.so/docs/model-yaml/ is up to date.
    3. To apply update user need to stop running models and re run chat to load new configuration
  5. namchuai commented on Sep 5, 2024

    @namchuai
    Contributor

    Model Downloading

    1. What happens when model download fails halfway (e.g. internet disconnected)?
    • We don't support resume failed/pause download.
    • If model download is failed, its <model_id>.yaml (inside <data_folder>/models) won't be created and won't display in our model list.
    1. How do we detect models? e.g. if users "import models locally" would it still work
    • Scan for <model_id>.yaml file inside <data_folder>/models.
    Screenshot 2024-09-05 at 08 52 40
    1. How do we version models?
    • Currently, we using a field version inside yaml file to store the model's version. Please not that we don't have logic to support model update at the moment. For now, Version is just for display purpose.

    3.1. If we update our model.yaml, or remote model binary in the HF branch, will download still work? Or will "redownloading/updating" fail due to "model exists"

    • It will display model exists.
    1. Letting users do cortex models update is currently out of scope right?
    • Yes, it is. We haven't work on this.
  6. locked and limited conversation to collaborators on Sep 5, 2024
  7. converted this issue into a discussion #1113 on Sep 5, 2024
  8. moved this from Need Investigation to Completed in Menloon Sep 5, 2024
  9. moved this from Completed to Discontinued in Menloon Sep 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions