Context: I've been developing a local RAG system with FastAPI using Ollama as the brains for embeddings and response creation. I've implemented asyncio within to avoid overloading the API but still have found an edge case where the process keeps running even after an error occurs blocking the API, so to test again after reviving the FastAPI server I need to open Windows Task Manager and kill the Ollama process to be able to make new requests.
Desire: Having a endpoint within the API that cancels any ongoing excecution and clears any queue pending so it can be called when handling errors.
Previous issues: I started by checking open issues realted to this and found two (287 and 492) that asked for something similar, however they are at least 1y old and had no good responses.
Other suggestions: People suggest using the generate and chat endpoints with parameter keep_alive set to 0, however I believe this may not be the solution since it requires the request to be taken and addressed.
Context: I've been developing a local RAG system with FastAPI using Ollama as the brains for embeddings and response creation. I've implemented asyncio within to avoid overloading the API but still have found an edge case where the process keeps running even after an error occurs blocking the API, so to test again after reviving the FastAPI server I need to open Windows Task Manager and kill the Ollama process to be able to make new requests.
Desire: Having a endpoint within the API that cancels any ongoing excecution and clears any queue pending so it can be called when handling errors.
Previous issues: I started by checking open issues realted to this and found two (287 and 492) that asked for something similar, however they are at least 1y old and had no good responses.
Other suggestions: People suggest using the generate and chat endpoints with parameter keep_alive set to 0, however I believe this may not be the solution since it requires the request to be taken and addressed.