You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Throughout this tutorial, we've been working with OpenAI's remote models, however wouldn't it be nice if we could work
6
-
with models on our local machine (without incurring costs!)
6
+
with models on our local machine (without incurring costs)?
7
7
8
-
We could install models on our local machinewith Instructlab or Ollama, however in this case let's start with (arguably) the easiest way: Podman Desktop AI.
8
+
Podman Desktop is a GUI tool that helps with running and managing containers on our local machine, but it can also help with running AI models locally as well thanks to its AI Lab extension. Thanks to Quarkus and LangChain4j it then becomes trivial to start developing with these models. Let's find out how!
9
9
10
10
11
11
== Installing Podman Desktop AI
12
12
13
-
First, you must download and install the specific Ollama version on your operating system. https://ollama.com/download[The instructions can be found here, window="_blank"].
13
+
First, if you haven't yet, you must download and install Podman Desktop on your operating system. https://podman-desktop.io/downloads[The instructions can be found here, window="_blank"].
14
14
15
-
Once installed, go ahead and download the local model by running this command:
15
+
NOTE: For Windows/Mac users, if you can, give the podman machine at least 8GB of memory and 4 CPU (Generative AI Models are resource hungry!). The model will run with less resources, but it will be significantly slower.
16
16
17
-
[.console-input]
18
-
[source,bash]
19
-
----
20
-
ollama pull llama3:latest
21
-
----
17
+
Once installed, go ahead and start the application and go through the setup process. After that, you should see an "AI Lab" extension in the left menu. If you don't, you may need to install the extension first. For that, go to Extensions -> Catalog and install Podman AI Lab.
22
18
23
-
Now, let's run our modal locally:
19
+
image::podman-desktop-ai.png[]
24
20
25
-
[.console-input]
26
-
[source,bash]
27
-
----
28
-
ollama serve
29
-
----
21
+
Go ahead and click on it, and in the AI Lab, select the "Catalog"
22
+
23
+
image::podman-desktop-ai-catalog.png[]
24
+
25
+
You should now see a list of available AI Models choose from. You can also import different ones (eg. from Huggingface), but we will use one of the InstructLab models that are already available.
26
+
27
+
NOTE: If you haven't heard of https://developers.redhat.com/articles/2024/05/07/instructlab-open-source-generative-ai[Instructlab], it's an open source project for enhancing large language models (LLMs) used in generative artificial intelligence (gen AI) applications. You can even contribute to it yourself!
28
+
29
+
To start using the model, we'll first need to download it, so go ahead and do that with the download button image:podman-desktop-model-download.png[download button,30] next to the instructlab/merlinite-7b-lab-GGUF entry (this might take a little while).
30
+
31
+
Once downloaded, you can create a new model service by clicking on the rocket button image:podman-desktop-create-model-service.png[rocket button, 30] that will appear where you previously clicked the download button.
32
+
33
+
You will be taken to the "Creating Model Service" page where you can set the port that should be exposed for the service. Podman Desktop assigns a random available port by default, but let's set it to `*35000*` so we can remember more easily what the port is when we configure our Quarkus application.
After a few moments, your very own Model service will be running locally on your laptop! You an check the details on the Service details page, including some samples to test out the service with cURL (or even Java!).
38
+
39
+
Now it's time to go back to our Quarkus application.
30
40
31
-
== Creating a new project with the Ollama extension
41
+
NOTE: The InstructLab service uses the OpenAI protocol, so we can continue to use the quarkus-langchain4j-openai extension for this exercise
32
42
33
-
The Ollama extension isn't compatible with other extensions we have used in this tutorial, so we'll create a new project.
43
+
44
+
45
+
== Creating a new project
46
+
47
+
Some of the exercises we completed previously are not compatible with non-OpenAI models, so let's create a fresh new project
return assistant.chat("Can you explain why the earth is flat?");
148
+
public String prompt() {
149
+
// feel free to update this message to any question you may have for the LLM.
150
+
String message = "Generate a class that returns the square root of a given number";
151
+
return assistant.chat(message);
119
152
}
120
153
}
121
154
----
122
155
123
156
== Invoke the endpoint
124
157
125
-
You can check your prompt implementation by pointing your browser to http://localhost:8080/earth/flat[window=_blank]
158
+
Let's ask our model to create a class that returns the square root of a given number:
159
+
160
+
You can check your prompt implementation by pointing your browser to http://localhost:8080/instructlab[window=_blank]
126
161
127
162
You can also run the following command:
128
163
129
164
[.console-input]
130
165
[source,bash]
131
166
----
132
-
curl localhost:8080/earth/flat
167
+
curl http://localhost:8080/instructlab
133
168
----
134
169
135
-
An example of output (it can vary on each prompt execution):
170
+
An example of output (remember, your result will likely be different):
136
171
137
172
[.console-output]
138
173
[source,text]
139
174
----
140
-
I think there may be a misunderstanding here!
175
+
Here is a simple Java class to calculate the square root of a given number using the built-in `Math` class in Java:
176
+
177
+
```java
178
+
public class SquareRootCalculator {
179
+
public static void main(String[] args) {
180
+
int num = 16; // square root of 16 is 4.0
181
+
double result = Math.sqrt(num);
182
+
System.out.println("Square root of " + num + ": " + result);
183
+
}
184
+
}
185
+
```
186
+
187
+
Alternatively, if you want to handle negative numbers or non-integer inputs, you can use the `Math.sqrt()` function directly:
188
+
189
+
```java
190
+
public class SquareRootCalculator {
191
+
public static void main(String[] args) {
192
+
double num = -16; // square root of -16 is -4.0
193
+
double result = Math.sqrt(num);
194
+
System.out.println("Square root of " + num + ": " + result);
195
+
}
196
+
}
197
+
```
198
+
199
+
This will allow you to calculate the square root of any given number, positive or negative, and handle non-integer inputs.
200
+
----
201
+
202
+
NOTE: depending on your local resources, this might take a up to a few minutes. If you run into timeouts,
203
+
you can try changing the `quarkus.langchain4j.openai.timeout` value in the application.properties file.
204
+
If you're running on Mac/Windows, you could also try to give the podman machine more CPU/Memory resources.
205
+
206
+
Notice that (at least in our case) the LLM responded with a Java class, since we provided in the SystemMessage that the
207
+
LLM should respond as if they were a Java engineer.
141
208
142
-
Actually, the scientific consensus is that the Earth is an oblate spheroid, meaning it's slightly flattened at the poles and bulging at the equator. The evidence from various fields of science, including astronomy, geology, and physics, all point to the fact that our planet is indeed round.
209
+
== Going further
143
210
144
-
Here are some reasons why we know the Earth is not flat:
211
+
Feel free to play around with the different models Podman Desktop AI Lab provides. You will notice that some are faster
212
+
than others, and some will respond better to specific questions than others, based on how they have been trained.
145
213
146
-
1. **Ships disappearing over the horizon**: When a ship sails away from an observer on the shore, it will eventually disappear from view as it sinks below the horizon due to the curvature of the Earth.
147
-
----
214
+
NOTE: If you want to help improve the answers generated by the InstructLab model, feel free to https://github.com/instructlab/community/blob/main/README.md[contribute to the project].
0 commit comments