OLLAMA.md
Learn how Ollama fits into the processing pipeline and how to swap in other providers.
This project uses Ollama to run local AI models during image processing.
Running models locally makes it easy to experiment with prompts, compare different models, and process images without sending every request to a hosted API.
The overall architecture isn't tied to Ollama, though. It's simply the provider used throughout this project. You could replace it with another service while keeping the rest of the processing pipeline almost exactly the same.
There are a few reasons I chose Ollama for this project.
Prompt engineering usually involves a lot of iteration.
You'll probably:
Running models locally makes those experiments quick and inexpensive.
Generating metadata for hundreds or thousands of images can become expensive if every request goes to a hosted API.
With local models you can experiment freely without thinking about token usage.
Once you're happy with the results, you could always move to a hosted provider if it better fits your application.
Some projects can't send images to external services.
Running models locally gives you another option.
Whether privacy is important depends entirely on your application, but it's nice to have that flexibility.
One of the most important design decisions in this project is separating image understanding from metadata generation.
Rather than asking one model to do everything, the pipeline looks like this:
Image
│
▼
Vision Model
│
Natural Language Description
│
▼
Instruction Model
│
Structured JSON
│
▼
MongoDB DocumentThis approach consistently produced better results during development.
The vision model focuses on understanding the image.
The instruction model focuses on producing predictable, structured output.
Keeping those responsibilities separate also makes prompts much easier to evolve over time.
The exact models may change as new versions become available, but the overall architecture remains the same.
Configuration lives in:
tools/process/config.jsThis makes it easy to swap models without changing the rest of the processing pipeline.
One thing worth pointing out is that Ollama is only used during processing.
Once metadata has been generated and stored in MongoDB, the application itself doesn't depend on Ollama.
Searching images only uses:
That separation keeps the runtime application lightweight.
Although this project uses Ollama, the processing pipeline isn't tied to it.
The same architecture would work with providers such as:
The important part isn't the provider.
It's the pipeline:
After building this project, here are a few things I'd recommend.
Don't ask one prompt to solve every problem.
Breaking the work into smaller, focused prompts generally produces more consistent results.
Prompt engineering evolves over time.
Storing prompt versions alongside generated metadata makes it much easier to understand where a particular result came from and to compare outputs as your prompts improve.
Avoid hardcoding model names throughout your application.
Keeping them in a single configuration file makes experimentation much easier.
Although this project focuses on image analysis, the same pattern works well for many other kinds of data.
The input changes.
The architecture stays remarkably similar.
For a complete walkthrough of how Ollama fits into this project, watch the MongoDB tutorial: