Skip to content

cld2labs/llama-3.1-8b-instruct#98

Merged
AhmedSeemalK merged 11 commits into
opea-project:mainfrom
cld2labs:cld2labs/llama-3.1-8b-instruct
Jun 23, 2026
Merged

cld2labs/llama-3.1-8b-instruct#98
AhmedSeemalK merged 11 commits into
opea-project:mainfrom
cld2labs:cld2labs/llama-3.1-8b-instruct

Conversation

@arpannookala-12

Copy link
Copy Markdown
Contributor

Summary

  • Adds model card for llama-3.1-8b-instruct (Meta) under third_party/Dell/model-deployment/llama-3.1-8b-instruct/
  • Adds Helm-based deployment guide for deploying llama-3.1-8b-instruct via vLLM on Gaudi and CPU (Xeon) with Keycloak OIDC and APISIX ingress

@alexsin368
alexsin368 self-requested a review April 29, 2026 04:18
@alexsin368

Copy link
Copy Markdown
Collaborator

Model deployment works. Testing inference is showing a Gateway Timeout error.

vLLM pod is fine, but ingress-nginx-controller is giving an upstream timeout:

2026/05/18 23:31:52 [error] 265195#265195: *10094624 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.17.23.1, server: api.example.com, request: "POST /Llama-3.1-8B-Instruct-vllmcpu/v1/completions HTTP/2.0", upstream: "http://10.233.104.80:9080/Llama-3.1-8B-Instruct-vllmcpu/v1/completions", host: "api.example.com"
172.17.23.1 - - [18/May/2026:23:31:52 +0000] "POST /Llama-3.1-8B-Instruct-vllmcpu/v1/completions HTTP/2.0" 504 160 "-" "curl/7.81.0" 1233 60.001 [auth-apisix-auth-apisix-gateway-80] [] 10.233.104.80:9080 0 60.000 504 b90b75191948d3bf2aff518dc7b72510

Comment thread third_party/Dell/model-deployment/llama-3.1-8b-instruct/xeon-deployment.md Outdated
@alexsin368

Copy link
Copy Markdown
Collaborator

inference is functional after increasing ingress and APISIX timeout to 300s

Comment thread third_party/Dell/model-deployment/troubleshooting.md Outdated
Comment thread third_party/Dell/model-deployment/troubleshooting.md Outdated
Comment thread third_party/Dell/model-deployment/troubleshooting.md Outdated
AhmedSeemalK
AhmedSeemalK previously approved these changes Jun 19, 2026
mdfaheem-intel
mdfaheem-intel previously approved these changes Jun 19, 2026
@HarikaDev296
HarikaDev296 dismissed stale reviews from mdfaheem-intel and AhmedSeemalK via ec99583 June 22, 2026 13:46
@HarikaDev296
HarikaDev296 force-pushed the cld2labs/llama-3.1-8b-instruct branch from 7aef9e1 to b376257 Compare June 22, 2026 14:23
alexsin368 and others added 11 commits June 22, 2026 09:24
Signed-off-by: alexsin368 <alex.sin@intel.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
…ell EI

Signed-off-by: arpannookala-12 <ganesh.arpan.nookala@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
Co-authored-by: alexsin368 <109180236+alexsin368@users.noreply.github.com>
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
@AhmedSeemalK
AhmedSeemalK merged commit 4764291 into opea-project:main Jun 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants