My solution is to ask llm to return result in some format, like markdown code block, for example. And then process model output line by line to check if line matches prefix/suffix patterns to change state of simple parser state machine. All data before and including prefix and after and including suffix will be dropped. But line-by-line processing is a key here.

I’m not sure what you mean by filtering results, can you give an example?

Not all models can reply with code without additional quotes and explanation. So to use this reply in code file we need to filter code part only to insert it into code buffer.

Thank you. Will see if I can switch to your package as a backend.

Very interesting offer. I need to see if you package can provide all low level functionalities to implement my ideas. Are you package provide streaming with ollama llm provider? Can I filter results with streaming enabled? Can I insert generation results to any buffer at any point with streaming enabled? Can I change model, temperature and system messages for implement agents-like functions? Thank you for response :)