Disclaimer
OK that title is truly clickbait.
We do not create our own large language model.
We do not even finetune a large language model.
We just persistently set it's parameters (or some) to custome values, including the system prompt.
Usecase
ComfyUI excels at creating absolut realistic images with the Fluxmania model (in conjunction with some other LORAs). To achive the best, realistic solution it is not enough to use a simple prompt like for example
amalfi coast sunset
No, you rather have to be very specific (because promting matters):
Some golden standards (like f.e. "the rule of thirds") simply needs to be followed. We need a certain camera model and need to adjust its shutter speed (to whatever fits best for the given scene). And some other things.
In software development this would be called boilerplate.
With the help of a LLM you could create some kind of template engine that turns a simple prompt like "amalfi coast sunset" into a prompt that includes all the relevant details.
Whenever you feed a LLM with the template below
Generate a prompt, start with [Type of photo, e.g., Photo, Portrait photo, Landscape photo, etc.] of a [subject/scene] in [lighting conditions, e.g., golden hour, soft natural light, etc.], captured with a [camera model] camera and a [lens model] lens. Set the aperture to [f-stop] for [depth of field effect, e.g., shallow depth of field for a blurred background, wide depth of field for sharp focus]. The composition should focus on [composition details, e.g., rule of thirds, centered, off-center]. Adjust the shutter speed to [shutter speed] to ensure sharpness. Set the white balance to [white balance setting, e.g., warm tones, neutral]. [Additional elements like reflections, shadows, rim lighting, etc.] can be used to add depth and emphasis to the subject.
Example
Portrait photo of a woman in soft natural light, captured with a Canon EOS R5 camera and a 50mm f/1.2L lens. Set the aperture to f/2.0 for shallow depth of field, creating a blurred background that emphasizes the subject. The composition should focus on the rule of thirds, with the subject slightly off-center. Adjust the shutter speed to 1/125 to ensure sharpness. Set the white balance to neutral for natural color tones. Soft shadows and rim lighting can be used to add depth and highlight the subject's features.
or
Landscape photo of a modern house during sunset, captured with a Nikon Z7 II camera and a 24-70mm f/2.8 lens. Set the aperture to f/8 for a wide depth of field, ensuring the entire scene from the house to the sky is in sharp focus. The composition should follow the rule of thirds, with the house positioned slightly off-center and the sunset filling the upper part of the frame. Adjust the shutter speed to 1/250 to maintain sharpness and capture the vibrant colors of the sunset. Set the white balance to warm tones to enhance the golden hues of the setting sun. Reflections of the sunset on windows and soft shadows can add depth to the image.
Generate the prompt only, no quotes at the beginning and at the end.
it will extend all your following simple prompts into detailed photographic prompts that are truly useful for ComfyUI.
Disadvantage: You have to save the prompt template somewhere as a TXT file. Everytime you start up your local running LLM (or start up another ChatGPT online session) you have to remember where you've stored this file, open it in notepad and paste its content into the chat session. Day by day by day...
Wouldn't it be nice if there would exist a LLM that exactly knows what you want so you don't have to repeat this over and over again?
Well, here we go!
Prerequisites
Ollama running localy on your system.
Docker running localy, hosting a Open WebUI container that interacts with your Ollama installation.
Open WebUI is (basically) your well known ChatGPT chat session, running on your own machine.
Docker Image Consideration:
On the Github project site for Open WebUI you will get this instruction for creating a docker container that hosts Open WebUI and interacts with your local Ollama instance using your GPU (and you should utilize your GPU, given it has at least 16GB of VRAM):
docker run -d -p 3000:8080 \\
--gpus all \\
--add-host=host.docker.internal:host-gateway \\
-v open-webui:/app/backend/data \\
--name open-webui \\
--restart always ghcr.io/open-webui/open-webui:cuda
This docker command does use persistant storage by using the -v option. So when you update your Open WebUI image all of your data (documents for RAG, saved API keys, created users, previous chat sessions, ...) will still exist. Make shure you use it.
You can now start Open WebUI via http://localhost:3000/ and do some basic initial settings (like creating the admin user).
Customize an existing LLM
Now it's time to "create our own LLM". Well, like said before, we do not really create a new model. Nevertheless the Ollama command itself is called "create".
Ollama lets you build a custom‑built version of any supported model by creating a Modelfile that tweaks the model’s settings or injects a new system prompt.
The command you’ll run is:
ollama create <model‑name> -f <Modelfile‑path>
Creating the Modelfile
The Modelfile is a plain‑text file that follows a simple syntax and has the name "modelfile" (no extension).
At a minimum, you need the FROM line that specifies the base model.
After that you can change parameters, add your own system prompt, or set resource limits.
We customize the gemma3:4b model here from Google, but you can use any model (so f.e. also gpt-oss:20b).
FROM gemma3:4b
# sets the temperature (higher is more creative, lower is more coherent)
PARAMETER temperature 0.675
# sets the context window size to 4096, this controls how many tokens the LLM can use as context to generate the next token
PARAMETER num_ctx 4096
# sets the system prompt
SYSTEM """
Generate a prompt, start with [Type of photo, e.g., Photo, Portrait photo, Landscape photo, etc.] of a [subject/scene] in [lighting conditions, e.g., golden hour, soft natural light, etc.], captured with a [camera model] camera and a [lens model] lens. Set the aperture to [f-stop] for [depth of field effect, e.g., shallow depth of field for a blurred background, wide depth of field for sharp focus]. The composition should focus on [composition details, e.g., rule of thirds, centered, off-center]. Adjust the shutter speed to [shutter speed] to ensure sharpness. Set the white balance to [white balance setting, e.g., warm tones, neutral]. [Additional elements like reflections, shadows, rim lighting, etc.] can be used to add depth and emphasis to the subject.
Example
Portrait photo of a woman in soft natural light, captured with a Canon EOS R5 camera and a 50mm f/1.2L lens. Set the aperture to f/2.0 for shallow depth of field, creating a blurred background that emphasizes the subject. The composition should focus on the rule of thirds, with the subject slightly off-center. Adjust the shutter speed to 1/125 to ensure sharpness. Set the white balance to neutral for natural color tones. Soft shadows and rim lighting can be used to add depth and highlight the subject's features.
or
Landscape photo of a modern house during sunset, captured with a Nikon Z7 II camera and a 24-70mm f/2.8 lens. Set the aperture to f/8 for a wide depth of field, ensuring the entire scene from the house to the sky is in sharp focus. The composition should follow the rule of thirds, with the house positioned slightly off-center and the sunset filling the upper part of the frame. Adjust the shutter speed to 1/250 to maintain sharpness and capture the vibrant colors of the sunset. Set the white balance to warm tones to enhance the golden hues of the setting sun. Reflections of the sunset on windows and soft shadows can add depth to the image.
Generate the prompt only, no quotes at the beginning and at the end. Always respond in English, even if the original prompt may have been composed in another language.
"""
We will name our "new model" simply after our usecase, so we will create the "new" model "comfyprompt":
Result
Now we can enter "simply prompts" like for example
portrait old white man in his 70s
or this german prompt
kind. 7 jahre alt. braune gelockte haare. in seinem kinderzimmer. fenster, dahinter garten sichtbar.
and our customized model "comfyprompt" will generate the prompts according to our specified template in the system prompt. Since we told the model to always answer in english regardless of the language used in the user prompt, the result for the german prompt of course is in english:
And here is the result in ComfyUI:
The image itself:
Summary
- Pull a base model (
ollama pull <model>). - Write a
ModelfilewithFROMand optionalSETparameters or a custom prompt. - Run
ollama create <custom‑name> -f Modelfile. - Use your new model with
ollama run <custom‑name>.
This workflow gives you a lightweight way to tailor any Ollama‑supported model for a specific tone, style, or set of constraints without the overhead of full fine‑tuning.