You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Throughout this tutorial, we've been working with OpenAI's remote models, however wouldn't it be nice if we could work
6
-
with models on our local machine (without incurring costs!)
6
+
with models on our local machine (without incurring costs)?
7
7
8
-
We could install models on our local machinewith Instructlab or Ollama, however in this case let's start with (arguably) the easiest way: Podman Desktop AI.
8
+
Podman Desktop is a GUI tool that helps with running and managing containers on our local machine, but it can also help with running AI models locally as well thanks to its AI Lab extension. Thanks to Quarkus and LangChain4j it then becomes trivial to start developing with these models. Let's find out how!
9
9
10
10
11
11
== Installing Podman Desktop AI
12
12
13
-
First, you must download and install the specific Ollama version on your operating system. https://ollama.com/download[The instructions can be found here, window="_blank"].
13
+
First, if you haven't yet, you must download and install Podman Desktop on your operating system. https://podman-desktop.io/downloads[The instructions can be found here, window="_blank"].
14
14
15
-
Once installed, go ahead and download the local model by running this command:
15
+
Once installed, go ahead and start the application and go through the setup process. After that, you should see an "AI Lab" extension in the left menu
16
16
17
-
[.console-input]
18
-
[source,bash]
19
-
----
20
-
ollama pull llama3:latest
21
-
----
17
+
image::podman-desktop-ai.png[]
22
18
23
-
Now, let's run our modal locally:
19
+
Go ahead and click on it, and in the AI Lab, select the "Catalog"
24
20
25
-
[.console-input]
26
-
[source,bash]
27
-
----
28
-
ollama serve
29
-
----
21
+
image::podman-desktop-ai-catalog.png[]
22
+
23
+
You should now see a list of available AI Models choose from. You can also import different ones (eg. from Huggingface), but we will use one of the InstructLab models that are already available.
24
+
25
+
NOTE: If you haven't heard of https://developers.redhat.com/articles/2024/05/07/instructlab-open-source-generative-ai[Instructlab], it's an open source project for enhancing large language models (LLMs) used in generative artificial intelligence (gen AI) applications. You can even contribute to it yourself!
26
+
27
+
To start using the model, we'll first need to download it, so go ahead and do that with the download button image:podman-desktop-model-download.png[download button,30] next to the instructlab/merlinite-7b-lab-GGUF entry (this might take a little while).
28
+
29
+
Once downloaded, you can create a new model service by clicking on the rocket button image:podman-desktop-create-model-service.png[rocket button, 30] that will appear where you previously clicked the download button.
30
+
31
+
You will be taken to the "Creating Model Service" page where you can set the port that should be exposed for the service. Podman Desktop assigns a random available port by default, but let's set it to `*35000*` so we can remember more easily what the port is when we configure our Quarkus application.
After a few moments, your very own Model service will be running locally on your laptop! You an check the details on the Service details page, including some samples to test out the service with cURL (or even Java!).
36
+
37
+
Now it's time to go back to our Quarkus application.
30
38
31
-
== Creating a new project with the Ollama extension
39
+
NOTE: The InstructLab service uses the OpenAI protocol, so we can continue to use the quarkus-langchain4j-openai extension for this exercise
32
40
33
-
The Ollama extension isn't compatible with other extensions we have used in this tutorial, so we'll create a new project.
41
+
42
+
43
+
== Creating a new project
44
+
45
+
Some of the exercises we completed previously are not compatible with non-OpenAI models, so let's create a fresh new project
return assistant.chat("Can you explain why the earth is flat?");
146
+
public String prompt(@QueryParam("message") String message) {
147
+
return assistant.chat(message);
119
148
}
120
149
}
121
150
----
122
151
123
152
== Invoke the endpoint
124
153
125
-
You can check your prompt implementation by pointing your browser to http://localhost:8080/earth/flat[window=_blank]
154
+
Let's ask our model to create a class that returns the square root of a given number:
155
+
156
+
You can check your prompt implementation by pointing your browser to http://localhost:8080/instructlab?message=generate%20a%20class%20that%20returns%20the%20square%20root%20of%20a%20given%20number[window=_blank]
An example of output (it can vary on each prompt execution):
166
+
An example of output (remember, your result will likely be different):
136
167
137
168
[.console-output]
138
169
[source,text]
139
170
----
140
-
I think there may be a misunderstanding here!
171
+
Here is a simple Java class to calculate the square root of a given number using the built-in `Math` class in Java:
172
+
173
+
```java
174
+
public class SquareRootCalculator {
175
+
public static void main(String[] args) {
176
+
int num = 16; // square root of 16 is 4.0
177
+
double result = Math.sqrt(num);
178
+
System.out.println("Square root of " + num + ": " + result);
179
+
}
180
+
}
181
+
```
182
+
183
+
Alternatively, if you want to handle negative numbers or non-integer inputs, you can use the `Math.sqrt()` function directly:
184
+
185
+
```java
186
+
public class SquareRootCalculator {
187
+
public static void main(String[] args) {
188
+
double num = -16; // square root of -16 is -4.0
189
+
double result = Math.sqrt(num);
190
+
System.out.println("Square root of " + num + ": " + result);
191
+
}
192
+
}
193
+
```
194
+
195
+
This will allow you to calculate the square root of any given number, positive or negative, and handle non-integer inputs.
196
+
----
197
+
198
+
Notice that (at least in our case) the LLM responded with a Java class, since we provided in the SystemMessage that the
199
+
LLM should respond as if they were a Java engineer.
141
200
142
-
Actually, the scientific consensus is that the Earth is an oblate spheroid, meaning it's slightly flattened at the poles and bulging at the equator. The evidence from various fields of science, including astronomy, geology, and physics, all point to the fact that our planet is indeed round.
201
+
== Going further
143
202
144
-
Here are some reasons why we know the Earth is not flat:
203
+
Feel free to play around with the different models Podman Desktop AI Lab provides. You will notice that some are faster
204
+
than others, and some will respond better to specific questions than others, based on how they have been trained.
145
205
146
-
1. **Ships disappearing over the horizon**: When a ship sails away from an observer on the shore, it will eventually disappear from view as it sinks below the horizon due to the curvature of the Earth.
147
-
----
206
+
NOTE: If you want to help improve the answers generated by the InstructLab model, feel free to https://github.com/instructlab/community/blob/main/README.md[contribute to the project].
0 commit comments