Computer vision for Pokémon Battles: A YOLO and Tesseract-Based System for Automated Recognition and Gameplay Analysis
DOI:
https://doi.org/10.26439/interfases2025.n022.8270Keywords:
Computer Vision, Pokémon, YOLO, Tesseract, OCRAbstract
Pokémon Double Battles present a complex decision-making environment that has traditionally relied on manual data analysis. This paper introduces an automated system leveraging computer vision and deep learning to extract structured gameplay data from battle footage. Our approach integrates You Only Look Once (YOLO) for Pokémon sprite recognition along with Tesseract-based optical character recognition (OCR) for extracting move and status text. The study introduces a custom-built image dataset generated through the augmentation of publicly available Pokémon sprites, which is then used to train a YOLO model for sprite recognition. The system was tested across multiple controlled and real-world gameplay scenarios, achieving high accuracy in Pokémon recognition and action tracking. Additionally, a JSON-based gameplay notation system is proposed to structure battle sequences, thus improving analysis and strategic review. The results demonstrate the feasibility of AI-driven gameplay analysis, with potential applications for competitive players, game analysts, and developers. Given its exploratory nature, this study focuses on technical feasibility rather than statistical generalisation. Future work includes expanding the dataset, improving OCR performance, and enabling real-time processing to support broader practical use.
Downloads
References
ASUSTeK Computer. (n.d.). Zenbook 14 (UM425UA). Retrieved August 27, 2024, from https://www.asus.com/laptops/for-home/zenbook/zenbook-14-um425-ua/
Bazan, D., Casanova, R., & Ugarte, W. (2024). Use of custom videogame dataset and YOLO model for accurate handgun detection in real-time video security applications [Manuscript in preparation]. Universidad Peruana de Ciencias Aplicadas.
Cheng, Y. (2023). Character detection in first person shooter game scenes using YOLO-v5 and YOLO-v7 networks. In 2023 2nd International Conference on Data Analytics, Computing and Artificial Intelligence (ICDACAI) (pp. 825-831). IEEE. https://doi.org/10.1109/icdacai59742.2023.00160
El-Nasr, M. S., Drachen, A., & Canossa, A. (2016). Game analytics. Springer.
Elgato. (2024). 4K X. Retrieved August 19, 2024, from https://www.elgato.com/uk/en/p/game-capture-4k-x
Hallett, P., Adickes, Z., Marttinen, C., Vohra, S., & Pezzè, A. (2024). PokéAPI. Retrieved August 25, 2024, from https://github.com/PokeAPI/pokeapi
HEROZ, I. (2024, February 9). HEROZ、将棋AIの技術を生かし、ポケモンバトルに特化したゲーム演出AI 「Pokémon Battle Scope」を株式会社ポケモンと共同開発 「ポケモン竜王戦2024」ゲーム部門の配信画面に初導入 [HEROZ, leveraging its Shogi AI technology, has jointly developed with The Pokémon Company a game presentation AI specialized for Pokémon battles called ‘Pokémon Battle Scope,’ which is being introduced for the first time on the broadcast screen of the game division at the ‘Pokémon Ryuo Championship 2024]. https://heroz.co.jp/release/2024/02/09_press01-3/
Hu, S., Huang, T., & Liu, L. (2024). PokéLLMon: A human-parity agent for Pokémon battles with large language models. arXiv preprint arXiv:2402.01118. https://doi.org/10.48550/arXiv.2402.01118
Jocher, G., Chaurasia, A., & Qiu, J. (2024). Ultralytics YOLO (Version 8.2.82) [Software]. Ultralytics. Retrieved from https://github.com/ultralytics/ultralytics
Lladó Herrera, M. R. (2024). Computer vision in gaming: Analysing Pokémon battles [Unpublished master’s dissertation]. The University of Manchester.
Luo, G. (2024). Pokémon Showdown [Computer software]. Retrieved August 25, 2024, from https://github.com/smogon/Pokemon-Showdown
Moussaoui, H., Akkad, N. E., Benslimane, M., El-Shafai, W., Baihan, A., Hewage, C., & Rathore, R. S. (2024). Enhancing automated vehicle identification by integrating YOLO v8 and OCR techniques for high-precision license plate detection and recognition. Scientific Reports, 14(1), 14389. https://doi.org/10.1038/s41598-024-65272-1
Norström, L. (2019). Comparison of artificial intelligence algorithms for Pokémon battles [Master’s thesis, Chalmers University of Technology]. Chalmers Open Digital Repository. https://hdl.handle.net/20.500.12380/300015
OBS Project. (2024). OBS Studio. Retrieved August 19, 2024, from https://obsproject.com/
Pokémon Database. (2024). Pokémon sprite archive [Database]. Retrieved August 24, 2024, from https://pokemondb.net/sprites
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779-788. https://doi.org/10.1109/cvpr.2016.91
Simoes, D., Reis, S., Lau, N., & Reis, L. P. (2020). Competitive deep reinforcement learning over a Pokémon battling simulator. In 2020 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). IEEE.
Smith, R. (2007). An overview of the Tesseract OCR engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007). IEEE. https://doi.org/10.1109/ICDAR.2007.4376991
Takahiro, H., & Tomohiro, T. (n.d.). Amaze the world by the power of AI. HEROZ, Inc. Retrieved August 13, 2024, from https://heroz.co.jp/en/company/
Tesseract OCR. (n.d.-a). Improving the quality of the output. Retrieved August 16, 2024, from https://tesseract-ocr.github.io/tessdoc/ImproveQuality.html
Tesseract OCR. (n.d.-b). Tesseract User Manual. Retrieved August 16, 2024, from https://tesseract-ocr.github.io/tessdoc/
Thapliyal, T., Bhatt, S., Rawat, V., & Maurya, S. (2023). Automatic license plate recognition (ALPR) using YOLOv5 model and Tesseract OCR engine. In 2023 First International Conference on Advances in Electrical, Electronics and Computational Intelligence (ICAEECI). IEEE.
The Pokémon Company. (2024a). 2024 Pokémon North America International Championships. Retrieved August 24, 2024, from https://www.pokemon.com/us/play-pokemon/internationals/2024/north-america/about
The Pokémon Company. (2024b). Pokémon in figures. Retrieved August 6, 2024, from https://corporate.pokemon.co.jp/en/aboutus/figures/
The Pokémon Company. (2024c). Video game rules, formats, & penalty guidelines. Retrieved August 6, 2024, from https://www.pokemon.com/static-assets/content-assets/cms2/pdf/play-pokemon/rules/play-pokemon-vg-rules-formats-and-penalty-guidelines-en.pdf
The Pokémon Company. (2024d). With AI, now everybody can totally enjoy battle watching. The Pokémon Company. Retrieved August 5, 2024, from https://corporate.pokemon.co.jp/en/topics/detail/118.html
The Pokémon Company International. (2024). About us. The Pokémon Company International. Retrieved August 13, 2024, from https://corporate.pokemon.com/en-us/about/
Ultralytics. (2024). This is Ultralytics. Retrieved August 26, 2024, from https://www.ultralytics.com/about
Yannakakis, G. N., & Togelius, J. (2018). Artificial intelligence and games (Vol. 2). Springer.
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under an Attribution 4.0 International (CC BY 4.0) License. that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
Last updated 03/05/21
