Capabilities of Gemini, GPT, and DeepSeek for transportation in Dutch: Assessing domain-specific and underrepresented language challenges
DOI:
https://doi.org/10.59490/ejtir.2026.26.4.8374Keywords:
transportation , large language model , underrepresented language , domain-specific task, GPT, gemini, DeepSeekAbstract
Large Language Models (LLMs) have showcased remarkable progress in language comprehension and interactive decision-making, primarily in English and across general tasks. However, their capabilities in domain-specific reasoning and underrepresented languages remain underexplored. This study investigates the performance of LLMs in the Dutch language within the transportation domain, with a focus on cross-lingual generation and multimodal input processing. It aims to understand LLM's model behavior in the transportation domain to help select appropriate foundation models for further fine-tuning on specific tasks when training from scratch is not feasible due to data availability or high costs. We evaluate the performance of nine different LLMs from three prominent open- and closed-source LLM families: Gemini, GPT, and DeepSeek, using 972 questions distributed across six datasets.
These datasets vary in complexity, input modality, and in English or Dutch to comprehensively assess the models. Our findings show that while LLMs generally perform better in English, several models demonstrate strong cross-lingual capabilities in Dutch. The DeepSeek R1 and GPT-4o models perform relatively well in both languages, while Gemini 1.5-Pro struggles with Dutch content. Additionally, LLMs exhibit a solid understanding of the context of transportation and regulation by effectively interpreting both information from images and text to answer domain-specific questions. However, they still lag behind the performance of elementary children in comprehensive transportation-related tasks, suggesting that LLM reasoning under zero-shot conditions may be limited. This study underscores the potential of LLMs in domain-specific applications and underrepresented languages and supports informed selection of foundation models for fine-tuning in these context settings.
Downloads
References
Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., Ahmed, M., Bali, K., & Sitaram, S. (2023). MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 4232–4267). https://doi.org/10.18653/v1/2023.emnlp-main.258
Alghisi, S., Rizzoli, M., Roccabruna, G., Mousavi, S. M., & Riccardi, G. (2024). Should we fine-tune or RAG? Evaluating different techniques to adapt LLMs for dialogue. In S. Mahamood, N. L. Minh, & D. Ippolito (Eds.), Proceedings of the 17th International Natural Language Generation Conference (pp. 180–197). https://doi.org/10.18653/v1/2024.inlg-main.15
Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F. (2024). Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning. Journal of Computational Social Science, 8, 17. https://doi.org/10.1007/s42001-024-00345-9
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Nguyen, V. Q., Do, T. T., Xu, Y., & Fung, P. (2023). A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 1, 675–718. https://doi.org/10.18653/v1/2023.ijcnlp-main.45
Buscaldi, D., & Rosso, P. (2023). How good is NLLB for low-resource languages? A study on the Genoese language. CEUR Workshop Proceedings, 3596. https://ceur-ws.org/Vol-3596/short5.pdf
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P., & Sun, L. (2025). A survey of AI-generated content (AIGC). ACM Computing Surveys, 57(5), 1–38. https://doi.org/10.1145/3704262
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., & Xie, X. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45. https://doi.org/10.1145/3641289
Chen, C., Hsieh, K., Huang, K., & Lai, H. (2024). Comparing vision-capable models, GPT-4 and Gemini, with GPT-3.5 on Taiwan's pulmonologist exam. Cureus, 16(8), e67641. https://doi.org/10.7759/cureus.67641
Common Crawl Foundation. (n.d.). Statistics of Common Crawl monthly archives by commoncrawl. Retrieved July 29, 2024, from https://commoncrawl.github.io/cc-crawl-statistics/plots/languages
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023). ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health Frontiers in Public Health, 11, Article 1166120. https://doi.org/10.3389/fpubh.2023.1166120
Deep Infra. (2025). DeepSeek V-3. Retrieved April 23, 2025, from https://deepinfra.com/deepseek-ai/DeepSeek-V3
DeepSeek API Docs. (2025). DeepSeek-R1 Release. Retrieved April 22, 2025, from https://api-docs.deepseek.com/news/news250120
DeepSeek. (2025). DeepSeek AI: chat, models & independent guides. Retrieved April 20, 2025, from https://deepseek.ai/
Dey, K., Tarannum, P., Hasan, M. A., Razzak, I., & Naseem, U. (2024). Better to ask in English: Evaluation of large language models on English, low-resource and cross-lingual settings. arXiv. https://doi.org/10.48550/arXiv.2410.13153
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M. A., Al-Busaidi, A. S., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., . . . Wright, R. (2023). Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, Article 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
Fan, L., Li, L., Ma, Z., Lee, S., Yu, H., Hemphill, L. (2024). A Bibliometric Review of Large Language Models Research from 2017 to 2023. ACM Transactions on Intelligent Systems and Technology, 15(5), 1–25. https://doi.org/10.1145/3664930
Gemini Team. (2023). Gemini: A family of highly capable multimodal models. arXiv. https://doi.org/10.48550/arXiv.2312.11805
Guo, X., Zhang, Q., Jiang, J., Peng, M., Zhu, M., Yang, H.F. (2024). Towards explainable traffic flow prediction with large language models. Communications in Transportation Research, 4(100150). https://doi.org/10.1016/j.commtr.2024.100150
Hammarström, H. (2016). Linguistic diversity and language evolution. Journal of Language Evolution, 1(1), 19–29. https://doi.org/10.1093/jole/lzw002
Knill, K., & Young, S. (1997). Hidden Markov models in speech and language processing. In S. Young & G. Bloothooft (Eds.), Corpus-based methods in language and speech processing (pp. 27–68). Springer. https://doi.org/10.1007/978-94-017-1183-8_2
Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y. (2022). Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Advances in neural information processing systems (Vol. 35, pp. 22199–22213). Neural Information Processing Systems Foundation. https://doi.org/10.52202/068431-1613
Lai, V., Ngo, N., Pouran Ben Veyseh, A., Man, H., Dernoncourt, F., Bui, T., Nguyen, T. (2023). ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 13171-13189). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-emnlp.878
Langhe, L.D., Maladry, A., Vanroy, B., Bruyne, L.D., Singh, P., Lefever, E., Clercq, O.D. (2024). Benchmarking Zero-Shot Text Classification for Dutch. Computational Linguistics in the Netherlands, 13, 63–90. https://www.clinjournal.org/clinj/article/view/172
Li, Y., Li, L., Wu, Z., Bing, Z., Xuanyuan, Z., Knoll, A.C., Chen, L. (2024). UnstrPrompt: Large Language Model Prompt for Driving in Unstructured Scenarios. IEEE Journal of Radio Frequency Identification, 8, 367-375. https://doi.org/10.1109/JRFID.2024.3367975
Liang, Y., Liu, Y., Wang, X., Zhao, Z. (2024). Exploring large language models for human mobility prediction under public events. Computers, Environment and Urban Systems 112(102153). https://doi.org/10.1016/j.compenvurbsys.2024.102153
Lin, X.V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., Pasunuru, R., Shleifer, S., Koura, P.S., Chaudhary, V., O’Horo, B., Wang, J., Zettlemoyer, L., Kozareva, Z., Diab, M., Stoyanov, V., Li, X. (2022). Few-shot Learning with Multilingual Generative Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 9019-9052). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.emnlp-main.616
Liu, R., Geng, J., Wu, A.J., Sucholutsky, I., Lombrozo, T., Griffiths, T.L. (2024). Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse. arXiv. https://doi.org/10.48550/ARXIV.2410.21333
Mo, B., Xu, H., Zhuang, D., Ma, R., Guo, X., Zhao, J. (2023). Large Language Models for Travel Behavior Prediction. arXiv. https://doi.org/10.48550/arXiv.2312.00819
Muller, B., Anastasopoulos, A., Sagot, B., Seddah, D. (2021). When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 448-462). Association for Computational Linguistics https://doi.org/10.18653/v1/2021.naacl-main.38
Muludi, K., Fitria, K.M., Triloka, J., Sutedi. (2024). Retrieval-Augmented Generation Approach: Document Question Answering using Large Language Model. International Journal of Advanced Computer Science and Applications, 15(3) . https://doi.org/10.14569/IJACSA.2024.0150379
Nazi, Z.A., Hossain, Md.R., Mamun, F.A. (2025). Evaluation of open and closed-source LLMs for low-resource language with zero-shot, few-shot, and chain-of-thought prompting. Natural Language Processing Journal, 10(100124). https://doi.org/10.1016/j.nlp.2024.100124
OpenAI. (2024a). Hello GPT-4o. Retrieved July 30, 2024. https://openai.com/index/hello-gpt-4o/
OpenAI. (2024b). GPT-4 Technical. arXiv. https://doi.org/10.48550/arXiv.2303.08774
OpenAI. (2022a). Introducing ChatGPT. Retrieved June 3, 2024. https://openai.com/blog/chatgpt
OpenAI. (2022b). DALL·E 2. Retrieved July 31, 2024. https://openai.com/index/dall-e-2/
Parvaiz, A., Nasir, E.S., Fraz, M.M. (2024). From Pixels to Prognosis: A Survey on AI-Driven Cancer Patient Survival Prediction Using Digital Histology Images. Journal of Imaging Informatics in Medicine, 37, 1728–1751. https://doi.org/10.1007/s10278-024-01049-2
Pichai, S., Hassabis, D. (2024). Our next-generation model: Gemini 1.5. Google. Revised July 3, 2024. https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., Shoham, Y. (2023). In-Context Retrieval-Augmented Language Models. Transactions of the Association for Computational Linguistics, 11, 1316–1331. https://doi.org/10.1162/tacl_a_00605
Ray, P.P. (2023). ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3, 121–154. https://doi.org/10.1016/j.iotcps.2023.04.003
Reynolds, D. (2009). Gaussian Mixture Models. In Li, S.Z., Jain, A. (Eds.) Encyclopedia of Biometrics (pp. 659-663). Springer. https://doi.org/10.1007/978-0-387-73003-5_196
Riaz, M.S., Cuenen, A., Janssens, D., Brijs, K., Wets, G. (2019). Evaluation of a gamified e-learning platform to improve traffic safety among elementary school pupils in Belgium. Personal and Ubiquitous Computing, 23, 931–941. https://doi.org/10.1007/s00779-019-01221-4
Sprague, Z.R., Yin, F., Rodriguez, J.D., Jiang, D., Wadhwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., Durrett, G. (2025). To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. In Proceedings of the International Conference on Learning Representations (ICLR 2025) (pp. 94118–94162). https://doi.org/10.48550/arXiv.2409.12183
Taal Unie. (n.d.). Feiten & cijfers – wat iedereen zou moeten weten over het Nederlands. Retrieved July 29, 2024. https://taalunie.org/informatie/24/feiten-cijfers
Vanroy, B. (2023). Language Resources for Dutch Large Language. arXiv. https://doi.org/10.48550/arXiv.2312.12852
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Łukasz, Polosukhin, I. (2017). Attention is All you Need. In Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (Eds.) Advances in Neural Information Processing Systems (pp. 6000-6010). Curran Associates, Inc. https://doi.org/10.48550/arXiv.1706.03762
Wang, Y., Zhao, Y. (2023). Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models. arXiv. https://doi.org/10.48550/ARXIV.2312.17661
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q.V., Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (pp. 24824-24837). Neural Information Processing Systems Foundation. https://doi.org/10.52202/068431-1800
Yan, H., Li, Y. (2025). Generative AI for Intelligent Transportation Systems: Road Transportation Perspective. ACM Computing Surveys, 57(12), 1-45. https://doi.org/10.1145/3719290
Zhang, S., Fu, D., Zhang, Z., Yu, B., Cai, P. (2023). TrafficGPT: Viewing, Processing and Interacting with Traffic Foundation Models. arXiv. https://doi.org/10.48550/ARXIV.2309.06719
Zhang, Z., Sun, Y., Wang, Z., Nie, Y., Ma, X., Li, R., Sun, P., Ban, X. (2025). Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Tasks. arXiv. https://doi.org/10.48550/ARXIV.2405.02357
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Thi M.D Tran, Davy Janssens, Geert Wets, Tom Brijs, Burcu Can, Jan Vuurstaek, Lien Aerts, Wim Ectors
This work is licensed under a Creative Commons Attribution 4.0 International License.
