Analyze a Soccer Game Using Tensorflow Object Detection and OpenCV Analyze a Soccer Game Using Tensorflow Object Detection and OpenCV
Introduction The world cup season is here and off to an interesting start. Whoever thought the reigning champions Germany would be... Analyze a Soccer Game Using Tensorflow Object Detection and OpenCV


The world cup season is here and off to an interesting start. Whoever thought the reigning champions Germany would be eliminated in the group stage 🙁

For the data scientist within you, lets use this opportunity to do some analysis on soccer clips. With the use of deep learning and opencv we can extract interesting insights from video clips. See example gif below of the game b/w Australia and Peru played where we can identify all the players + referees, the soccer ball and also predict which team the player is based on the color of their jersey. And all of this can be done real time.

Player detection and team prediction


You can find the code I used on my Github repo.

Overview of the steps

Tensorflow Object Detection API is a very powerful source for quickly building object detection models. If you are not familiar with this API, please see the following blogs from me that introduce the API and teach you how to build a custom model using the API.

Introduction to Tensorflow Object Detection API

Building a custom model using Tensorflow Object Detection API

The API provides pre-trained object detection models that have been trained on the COCO dataset. COCO dataset is a set of 90 commonly found objects. See image below of objects that are part of COCO dataset.

coco object categories


In this case, we care about classes — persons and soccer ball  – which are both parts of COCO dataset.

The API also has a big set of models it supports. See table below for reference.

Small subset of models supported by the API


The models have a trade off between speed and accuracy. Since I was interested in real time analysis, I chose SSDLite mobilenet v2.

Once we identify the players using the object detection API, to predict which team they are in we can use OpenCV which is a powerful library for image processing. If you are new to OpenCV please see the tutorial below:

OpenCV Tutorial

OpenCV allows us to identify masks of specific colours and we can use that to identify red players and yellow players. See example below of how OpenCV masking works to detect red colour in the image.

Prediction of red areas in an image

Deep Dive into the main steps

Now lets go into the code in detail.

If you are using the Tensorflow Object Detection API for the first time, please download the GitHub from this link. And install all the dependencies using these instructions.

If you don’t have OpenCV setup, please build it from source using this tutorial.

The main steps I followed are (please follow along in the jupyter notebook on my Github):

  • Load the SSDLite mobilenet model into a graph and load the list of classes that are part of COCO dataset
  • Open the video using cv2.VideoCapture(filename) and read each frame one by one
  • For each frame perform object detection using the loaded graph
  • The result that comes back from the SSDLite is each identified class along with its confidence score and bounding box prediction. So now identify all persons identified with confidence > 0.6 and crop them out.
  • Now you have each player extracted out. We need to read the color of their jersey to predict if they are an Australian player or a Peru player. This is done by the code block detect team. We first define the color ranges for red and blue colors. Then we use cv2.inRange and cv2.bitwise to create a mask of that color. To detect team, I count how many red pixels and yellow pixels were detected and what is the percent of that compared to total num of pixels in the cropped image.
  • Finally, all the code pieces are combined to run everything at the same time and display results using cv2.imshow

Conclusion and References

Awesome. So now you can see how simple combination of deep learning and OpenCV can produce interesting results. Now that you have this data, there are many ways to draw additional insights from it:

  1. With the camera angle at the Australian goal area, you can calculate how many Peru players are in the zone vs Australian players
  2. You can draw a heat map of each teams’ footprint — Example what are the areas which saw high occupancy from Peru team
  3. You can draw out the path of the goalkeeper

The Object Detection API also offers other models that are more accurate but slower. You can try those as well.

Other writingshttp://deeplearninganalytics.org/blog

PS: I have my own deep learning consultancy and love to work on interesting problems. I have helped several startups deploy innovative AI based solutions. Check us out at — http://deeplearninganalytics.org/.

If you have a project that we can collaborate on, then please contact me through my website or at priya.toronto3@gmail.com



Original Source

Priya Dwivedi

Priya Dwivedi

Priya Dwivedi has 10+ years experience as a data scientist. She now runs her own data analytics consultancy that builds deep learning models for Computer Vision and NLP problems. She has helped many startups deploy innovative AI based solutions. For more info please see the link — http://deeplearninganalytics.org/. If you are interested in collaborating with her then please contact her at priya.toronto3@gmail.com.