Anime Face Recognition

This is about a prototype I wrote in grad school for Dr. Kim’s computer vision class.  If anyone interested in computer vision ever decides to implement this as a real application, be sure to let me know so I can be impressed.

Power point slides: Download

Figure out who your waifu is now!

The Anime Face Recognition App

The anime face recognition app will maintain a database of anime characters along with their feature vectors.  Cartoon characters are drawn from simple shapes and colors, and usually do not have realistic shading. The features which distinguish between characters are most often hair color, eye color, hair style, and eye style. Here I discuss my thoughts on system which would automatically identify the features of an anime face and try to recognize what the character is. There are main tasks the system must perform:

  • Take a color image and compute a feature vector.
  • Take a feature vector and a database of character names, along with their average (precomputed) feature vectors, and return the name of the character who’s feature vector is closest to the input using some predefined metric.
  • Generate the database, either by automatically or manually computing the feature vectors of well known characters and recording the average feature vector.  The feature vectors for several well known characters could be computed, simply by choosing images returned by entering their names in Google image search. This would ensure that the feature vector actually represents the features of the character the way they are usually drawn, instead of possibly selecting an outlier. Additional information, such as standard deviation of the feature vectors could be collected.

I made the prototype with OpenCV in C++, so that it could run efficiently as a mobile application.  It still needs help from a human to select parts of the face.  The color input image is first converted to grayscale, where the black lines drawn by the artist appear very dark and the solid color regions become shades of grey. The greyscale image is converted to black and white using some intensity threshold. Then, the findContours function creates a new image, where white replaces the black edges. This image is dynamically recalculated and displayed to allow the user to adjust the threshold until it looks correct. Automatically determining the threshold from the standard deviation of the grey values would be more involved, of course.

One problem I found is that often two separate color regions become connected together when the black line dividing them is insufficiently thick. One approach to mitigate this is to dilate the white edges.

The user then manually clicks the face and hair regions. These clicks become seed points of the floodFill command, which creates a mask image by flooding the face regions with red and flooding the hair regions with blue. When all regions have been clicked, the feature vector is calculated by counting the number of blue and red pixels in the mask to determine the hair/face pixel ratio. Additionally, the positions of these points in the mask are used to look up color of the hair and face regions in the source image. The creates the feature vector, which will be compared to a database of feature vectors of well known characters along with their names. The advantage of this approach is that an mobile face recognition app does not need a huge database of character images, they only need to textually describe the features of each character. This allows the download to be small and efficient, without the need to connect to a remote database for every recognition.

Future work will likely focus on identifying the eyes. Eye color can be extracted by running the k-means algorithm with k=3, assuming there is one eye color an addition to black and white (usually a safe assumption. )  Opening Mark Crilley’s Mastering Manga book, I realized there are hundreds of types of anime eyes, so a bag of words approach could be appropriate.  First they eye images should be converted to greyscale to ensure we are only comparing the structure in this stage, not color. Also, all left eyes could be mirrored to make right eyes so we don’t have to store both.  The accuracy would skyrocket if eye identification could be implemented.

Other future work involves recognizing faces drawn in profile view. Here, the critical feature to look for is the nose. The one visible eye will likely look very different than in front and 3/4 view, so a separate bag of words will be needed for profile faces.  Identifying the hair color is simply a matter of detecting which large sections of color are directly above the face. Because many characters wear hats or other clothing, or are obscuring the hair with other body parts, being sure that all parts of the hair have been found is tricky. Of course, we restrict ourselves to the subset of connected components of the hair color we have already discovered, but this unfortunately can include parts of clothing and the background with the same color as the hair.

References

Takayama, Johan, and Nishita.  Face Detection and Face Recognition of Cartoon Characters Using Feature Extraction. Proceedings of the IIEEJ Image Electronics and Visual Computing Workshop 2012.

Leave a Reply

Your email address will not be published. Required fields are marked *