Abstract
We investigate the benefits of fine-tuning a vision-language model for extracting object relationships from images. By generating instruction samples from scene graph annotations, our method teaches the model to prioritize relevant relations and suppress spurious or hallucinated ones. Experiments show that domain-focused fine-tuning yields large accuracy gains in specialized settings.
Introduction
Understanding how objects relate to one another is fundamental to visual scene interpretation [2]. While modern vision-language models can describe scenes at a high level, they often struggle to extract specific object relationships from images. Without domain priors, they may surface irrelevant relations, miss important interactions, or hallucinate object-object relations that do not exist.
Supervised relation extraction systems provide accurate results, but they usually require extensive annotations, domain expertise, and model-specific training [3]. Recent developments in vision language models enable zero-shot relation reasoning via visual prompting [1], but these approaches frequently rely on elaborate prompt engineering and show inconsistent behavior across domains.
To address these limitations, in this work we investigate the benefits of fine-tuning a vision-language model for extracting object relationships from images. Starting from ground truth scene graph annotations, we generate instruction samples that pair raw frames and set of mark overlays with the corresponding object-object relations. These instructions are then used to fine-tune GEMINI model through lightweight adapter layers in the decoder.
We evaluate both the baseline and fine-tuned models on two domains: a mixed set of everyday scenes, and a focused coffee-making domain. Our results show that fine-tuning yields large performance gains when the evaluation domain matches the fine-tuning domain, with particularly strong improvements in precision and reduced hallucination of irrelevant relations. In contrast, the generalist baseline performs better in diverse scenes, underscoring a trade-off between broad generalization and domain specialization. This demonstrates that domain-focused fine-tuning can enable reliable relation extraction with modest data, even when base models already possess strong visual and linguistic capabilities.
Methods
Dataset
We construct our dataset using individual frames annotated with object categories and segmentation masks. Relevant object-object relations are annotated for each scene, such as “left hand holds mug” or “button part of coffee machine”. These annotations provide the foundation for generating instruction samples for model fine-tuning.
Instruction Generation
Ground truth annotations are converted into instruction sets. To keep the instruction sets diverse, frames are sampled in a uniform fashion from longer videos with temporally consistent annotations. Each instruction consists of:
- User message: Raw frame, frame with set-of-mark overlays, and a text prompt. In the text prompt, a mapping of object ids to the corresponding category is also included.
- Expected output: Ground truth object-object relations in that frame.
This process allows the model to learn both to recognize objects and to identify the correct relationships between them while suppressing irrelevant or spurious relations.
| Dataset | Training set | Validation set |
|---|---|---|
| Mixed scenes | 608 | 54 |
| Coffee-making | 4676 | 420 |
| Total | 5284 | 474 |
Table 1: Instruction set sizes used for fine-tuning and validation. The training set combines a mixed-domain dataset which includes various scenes e.g. outdoor, indoor, cooking, cleaning, pet caring and a focused coffee-making domain dataset containing videos of coffee making process, e.g. operating a capsule machine or a full automatic machine.
Model Fine-tuning
We fine-tune GEMINI 2.5 Flash and GEMINI 2.5 Flash Lite using adapter layers in the decoder, leveraging the instruction set to teach the model domain-specific object-object relations. The Vertex AI API is used for training and inference.
Results
We evaluate model performance by comparing predicted object relations against ground truth annotations at the frame level. A true positive is counted when a ground truth object pair for a given relation value exists anywhere in the predictions (regardless of the predicted relation value), a false negative occurs when a ground truth pair is not found in any predictions, and a false positive is counted when a predicted pair for a given relation value does not exist in the ground truth. This matching strategy focuses on object pair relation presence rather than the description. Precision, recall, and F1 scores are computed and reported separately for both datasets: Mixed scenes and Coffee-making. Metrics are aggregated across all frames and then aggregated to produce the final scores for each model.
| Mixed scenes | Precision | Recall | F1 score |
|---|---|---|---|
| GEMINI 2.5 Flash | 0.37 ± 0.01 | 0.62 ± 0.01 | 0.46 ± 0.01 |
| GEMINI 2.5 Flash * | 0.43 ± 0.01 | 0.74 ± 0.01 | 0.54 ± 0.01 |
| GEMINI 2.5 Flash Lite * | 0.50 ± 0.01 | 0.46 ± 0.01 | 0.48 ± 0.01 |
| GEMINI 3.0 Flash | 0.46 ± 0.01 | 0.60 ± 0.02 | 0.52 ± 0.01 |
| GEMINI 3.5 Flash | 0.59 ± 0.01 | 0.62 ± 0.00 | 0.61 ± 0.00 |
| Coffee-making | Precision | Recall | F1 score |
|---|---|---|---|
| GEMINI 2.5 Flash | 0.33 ± 0.01 | 0.43 ± 0.02 | 0.37 ± 0.02 |
| GEMINI 2.5 Flash * | 0.77 ± 0.00 | 0.88 ± 0.00 | 0.82 ± 0.00 |
| GEMINI 2.5 Flash Lite * | 0.84 ± 0.01 | 0.80 ± 0.00 | 0.82 ± 0.00 |
| GEMINI 3.0 Flash | 0.26 ± 0.01 | 0.31 ± 0.01 | 0.29 ± 0.01 |
| GEMINI 3.5 Flash | 0.48 ± 0.00 | 0.50 ± 0.00 | 0.49 ± 0.00 |
Table 2/3: Precision, recall, and F1 score for zero-shot and fine-tuned models on two evaluation splits. “1” indicates the best result per metric; “2” indicates the second best. On mixed scenes, the latest zero-shot model (Gemini 3.5 Flash) outperforms all fine-tuned variants, while on the domain-specific coffee-making split, fine-tuned models retain a substantial advantage.
Conclusion
In focused settings such as coffee making, fine-tuned models significantly outperform all zero-shot alternatives, including the latest Gemini 3.5 Flash, delivering outputs that consistently match the terminology and granularity expected by downstream systems. On the other hand, on general, open-world scenes, Gemini 3.5 Flash zero-shot outperforms our best fine-tuned model, showing that recognizing common relations between objects is increasingly handled at the foundation level.
This shows that fine-tuning a vision-language model on task-specific instructions transforms it into a domain expert for extracting object relationships from images. Even a modest fine-tuning set yields substantial gains when fine-tuning is carefully targeted to the task.
References
[1] Yang, J., Zhang, H., Li, F., Zou, X., Li, C., & Gao, J. (2023). Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441.
[2] Chang, X., Ren, P., Xu, P., Li, Z., Chen, X., & Hauptmann, A. (2021). A comprehensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 1-26.
[3] Li, H., Zhu, G., Zhang, L., Jiang, Y., Dang, Y., Hou, H., ... & Bennamoun, M. (2024). Scene graph generation: A comprehensive survey. Neurocomputing, 566, 127052.