Emotional recognition market research

The BBC research department awarded a $ 350,000 grant to a young specialist from the University of Texas to study the problem of recognizing facial expressions and human emotions. Research will be conducted over a period of three years. “In the future, when developing man-machine interfaces, more attention will be paid to the emotional component of a person,” says Yang Liu, the happy owner of the grant. “Machines will have to interact with a person based on his emotional state. This factor will play a large role in the creation of artificial intelligence systems in the near future. ”

Scientists have been working hard for many years to ensure that powerful computing systems are able to adequately capture and identify facial expressions. Nowadays there are great successes, and it is not only the perfection of the proposed methods and algorithms, but also the development of newer ones. Already, many companies offer proprietary software, so I wanted to learn more about the products on the recognition of emotions on the face of a person.
Under the cut a lot of text and pictures.

The recognition of emotions is part of a large reservoir of science, united in the names "Pattern Recognition" and "Processing of Visual Information". Nowadays, recognition technologies cease to be out of reach and a new trend is formed, entailing all those interested in the methods of identifying and processing found objects and their signs. They smoothly flow from the realm of fantasy to real life.

Existing emotion recognition systems

Of commercial solutions in the emotion recognition systems market, the most complete and more interesting to consider in the context of the emotion recognition task today is FaceReader from the Dutch company Noldus Information Technology.

1. FaceReader ™

• Company - developer: Noldus Information Technology (Netherlands)
• Product Readme Information
• Current Version: 4.0



The program can correctly interpret such facial expressions as “happy”, “sad”, “angry”, “surprised”, “frightened”, “displeased” and “neutral”, as can be seen in the figure. In addition, FaceReader is able to determine their age, gender and ethnicity by their faces. FaceReader does not need training and additional configuration.
The program implemented computer vision technology. In particular, this is the Active Template method, which consists in imposing a deformable pattern on the face image:



Also, the Active Appearance Model method is implemented, with the help of which you can create an artificial face model taking into account control points and surface details, and compare it with samples stored in memory.
The classification is done using neural network methods with a training set of 2,000 photos.

Program features:

• The average percentage of emotion recognition is 89%. It is higher for some emotions, lower for some;
• face tilt can be any in the plane, its system will detect;
• The program works with downloadable video in formats with MPEG1, MPEG2, XviD, DivX4, DivX5, DivX6, DV-AVI and uncompressed AVI codecs, and you can determine emotions by frame or completely when watching the entire video. Also, FaceReader can work with static images, as well as in real time, if the user has a webcam connected;
• the program is beautifully visualized: you can always see histograms, diagrams, percentages of expressed emotions. And on the timeline visible manifestations of microexpressions in a certain period of time;
• FaceReader generates two text files, one is the emotion log, and the other is static, for the unique visual data management system The Observer XT®, developed by the same company, associated with this program.

Disadvantages of the program:

• FaceReader is not trained to recognize children under 5 years old;
• If a person is wearing glasses, then recognition of emotions is inaccurate, or classification is not conducted;
• People with different skin colors are perceived differently by the system, the program is not fully adapted;
• Turned face is not detected.

2. eMotion Software and GladOrSad

• Company - developer: Visual Recognition (Netherlands)
• Product Readme Information
• Current Version: Unknown

The eMotion Software system is known for having recognized the emotions in the painting “Mona Lisa”. The result showed that she was 83% happy, 9% showed disgust, 6% fear and only 2% Mona Lisa was angry.
And the system is known for the fact that it is, in fact, the first commercial paid "boxed" solution. Along with this solution, the development team launched the site GladOrSad.com - respectively, Visual Recognition took the lead in the opening of a web resource dedicated to the online recognition of emotions.



The first known user of eMotion Software was Unilever, which introduced the recognition system into the ice cream maker - Unilever Share Happy. People are smiling at the machine, the machine gives free ice cream for smiles!

If a person shows emotions, smiles, frowns, or makes a face, thousands of small facial muscles are in work. The emotion recognition system, or ERS (Emotion-recognition system), creates a 3D model of the face, identifying 12 key areas, such as the corners of the eye and the corners of the mouth.
In these programs, the tracking algorithm identifies the same emotions, there are six of them: anger, sadness, fear, surprise, disgust and happiness, and also the seventh is their confusion.
The software is not particularly demanding on the computer in terms of technical characteristics. The details of the implementation of the algorithm is unknown, since technology is kept secret, brochures with explanations, I also did not find, unfortunately.

3. MMER_FEASy - the FacE Analysis System

• Company - developer: MMER-Systems (Germany)
• Product Readme Information
• Current Version: Unknown

Again, the development used the methodology of imposing on the face of a certain deformable mask, Active Appearance Model methodology, which allows you to calculate the necessary parameters in real time. The mask operation is shown below:



The system uses three plug-ins - MMER_Lab, MMER_GPU and MMER_Locate.
MMER_Locate ensures that the person is in the image, MMER_Lab classifies some features of this image, and MMER_GPU ensures the efficient operation of the entire system:



The program recognizes six basic emotions, also provides services for finding individuals of their age, gender and ethnicity. The system also identifies the person if at any time the reference photo has ever been loaded into the database.
Additional features of the program include connecting to other programs as a module for remote assistants, driver assistants, marketing research and home multimedia services.
The disadvantages of the program can be considered not the full coverage of the downloaded data, since You can only work with a webcam. Bad results and data upload, where you can view only the "avatarization" of the person, i.e. instead of a mask, this same face is substituted, but with a different facial expression.

4. FaceSecurity

• Company - developer: Cognitec (Germany)
• Product Readme Information
• Current Version: 4.6

This product consists of several manufactured components based on the FaceVACS SDK. It:
• FaceVACS-DBScan with Examiner
• FaceVACS-PortraitAcquisition
• FaceVACS-VideoScan

FaceVACS-DBScan with Examiner:

The development is designed to handle unique databases and data banks of any category of people, for example, employees.
This product is an embodiment of biometric identification by the standard of samples from the base.
In the new version of the program, the system uses the new comparison algorithm B5T8, along with the old A14T8, designed to improve the identified similarity.
In addition, the new component Examiner allows automatic image conversion for comparison in the gallery. This allows developed operators to view lists of potential partners while maintaining a full audit for each step in the process.
Also, this development helps investigators to identify persons in crime scenes by photography and video surveillance by comparing images of persons in the agency repository.
FaceVACS-Examiner also provides a set of tools that help the inspectorate to identify a person in a timely manner, which allows investigators to act in accordance with the search results in the minimum amount of time after the commission of a crime.

Features:

• Cluster configuration for multimillion processing of a database of people;
• Flexible and convenient list management, which allows you to sort it, view and filter;
• Package and interactive training, identification;
• Deep and flexible management of related data;
• Reconfigured logs;
• Pregeneration of databases (i.e., pre-fast pre-tuning before outputting the result);
• Available in many different search samples by database.

FaceVACS-PortraitAcquisition:

Creation and evaluation of digital portraits for photo identification documents. The operation of the component is illustrated in the figure:



This component simplifies the production of high-quality portraits for passport photos, driver's licenses and other documents that are suitable for face recognition.
The graphical user interface of the product is specially tailored to visually control and manipulate the processing of such trifles as frontal posture, uniform lighting, glasses, and squinting eyes. The software is specifically tuned to assess whether the image complies with the mandatory requirements and best practice recommendations of the ISO 19794-5 standard of the front image type. The program supports integration using web services (SOAP) to facilitate the issuance of the document.

Features:

• Full compliance with ISO 19794-5 standards;
• Reliable and automated information gathering process;
• Checking the frontal posture, glasses, uniform lighting, head size, image size, mouth opening, head rotation, checking for tinted glass, red eyes, front view of eyes, exposure, skin color, hot spots, sharpness;
• Convenient graphical user interface;
• Customizable parameters and thresholds;
• Supports the format of digital SLR cameras from Nikon (D5000) and Canon (EOS 1000D and 1100D EOS);
• Flexible pruning, if necessary;
• Custom image preview (size, image type, image format);
• Flexible integration with web services.

FaceVACS-VideoScan:

A new generation of computerized video surveillance automatically scans the incoming video stream, detects several faces and checks for possible matches in the “checklist.” If a match is found, operators are notified in real time.
The application includes not only the identification of unwanted people in public places, but also the identification of high-ranking clients.

Features:

• Automatic face tracking in real time on multiple video streams;
• In real time, the face is compared with the standards of the “checklist”;
• Registration with a still image or with a live stream of video in manual and automatic mode;
• Application of C ++ API and Web Services API;
• Scalability within a “checklist”: size, number of video streams, and number of visible faces in an image.

Cognitec features:

• Extremely high processing of comparisons according to samples (900 000 comparisons in the database per second on an average processor power);
• Integration with webcams, http - cameras, digital cameras, video cameras, as well as support for images in common formats;
• Bulk databases, integration with Oracle, IBM DB2, MSSQL Server.

Disadvantages:

• Calculations in almost all components are applicable only to frontal faces (deviations of 15 degrees are possible, but not more);
• Light plays a big role - for example, components in the shadow components do not recognize the image in the shadow.

Not to mention the high-profile web-solution based on FaceVACS-SDK and FaceVACS-DBScan, MyHeritage.com portal, where you can build your family tree on the basis of face recognition, as well as compare yourself with celebrities, face morphing, as well as recognize and mark yourself in the photo.
In addition to Web applications, Cognitec provides an API for digital billboards for displaying advertisements to target audiences.
As mentioned above, Cognitec is also involved in mechanical engineering, the systems of this company are used in automobiles for analyzing the faces of drivers and fellow travelers, as well as safety, for example, by detecting the position of the head, detecting a scattered look, detecting closed eyes.
Among the other companies, Cognitec has another advantageous distinguishing point: having its own mobile phone SDK.

5. Products of Affective Computing Research Group

• Company - developer: Affectiva (USA)
• Product Readme Information
• Current application software version: 1.0

Rosalind Picard's company, Affectiva, is best known for the Q-Sensor wearable biosensors supplied. But not only this rich company. There is a huge experience of introducing technologies among Affective computing, or emotional computing systems, the development has been going on since 1995. There are a lot of projects. This is the oldest group of developers involved in these technologies.
There is, for example, the AffQuake project based on the ID Software Quake 3 product. The point is that the game reacts to the player's emotional signals. The gamers are bombarded with sensors, and if it gets scary, the modified Quake receives the user's “physiological signals” and makes the player’s virtual embodiment fear the same way - it retreats in fear.
Or, for example, developed a toy "Emotional Tiger" (Affective Tigger). This robot can identify five emotional states of a child playing with it and express its emotion in response. If a child jumps, squeezes and kisses a toy cheerfully, then the emotion recognition system and sensory system of the “tiger” fix this physical effect, after which happiness is demonstrated: Affective Tigger laughs and smiles.
There is an interesting solution for the Web. It is an innovative product from Affectiva, a collection of data on the emotional state of people on the World Wide Web, Affdex. For the most part, it is used for marketing research.
One of the ways to recognize the emotional state of the face in these developments is to record in real time with subsequent computer analysis - methods of comparison with embedded samples (SURF and based on SIFT descriptors), as well as wavelet methods. The work of these methods is used in a program such as Pupeteer, which evaluates the behavior and emotional state of students. Demonstration of the program:



During the experiment on this method, the computer determines six basic emotions with 96 percent accuracy.
The decision is also remarkable because it recognizes, along with emotions, head movements, such as a nod or a swing, winding from side to side. The processes of Bayesian machine learning are used to classify emotions, as well as to calculate statistics and calculate mixed states, when it is impossible to express exactly which emotion prevails.
There is not much to say about the software itself, as the technologies are closed. Development is conducted in C ++, Objective C for the iPhone. Graphically drawn through the usual tools, such as timeline (or timeline), graphs and charts, as seen in the following figure:



Among the features, it can be noted that all solutions are well drawn and adapted to the customer, and of the drawbacks, perhaps, the greatest is the presence of a computer with a non-weak processor (above Core i5) for comfortable work with applications.

Company comparison and subtotals

Considered the decisions of only some players in this business. The rest of the companies provide their products that are intended for other tasks, but the systems they develop are somehow interesting, because they can be easily improved to the level of recognition of emotions. These are programs and solutions that perform such tasks as:
• verification of a person (security systems and access control);
• tracking and face tracking (CCTV systems);
• comparison of people in the image and likeness of his (search system);
• animation of the face and its transformation (morph systems);
• face transformation into 3D - models (modeling systems);
•determination of the race, age and sex of a person (gender classification system);
• much more.

Moreover, the majority of companies developing these programs provide their own toolkit (SDK - Software Development Kit) to any developer.
The key criterion for the use of such products is their cost , as well as the cost of the provided SDK. It ranges from $ 5 to $ 2,000. The most expensive products are unlikely to be used in small companies, where developments of this kind are not a condition for the work of the entire company.
Below, in the table, I provide a list of such programs and program complexes. Of course, the list will expand over time, but as of November 2011 it is relevant:


Actual activities of companies focused on web-development and technology. These are Google, Twitter, FaceBook and many others. Google Inc. uses face and emotion recognition technologies for a smarter search, Twitter assesses the mood of people writing in online blogs, FaceBook recently introduced a new feature (improvement) called Tagger to the public that automatically recognizes, marks and signs the faces of friends of a social network user.
The developers of Face.com are known for their development, as well as their own JavaScript SDK, which PhotoTagger uses.
Recently, this company introduced a technology that allows people to be identified from photographs published on Internet sites. The PhotoFinder program analyzes the digital images found on the pages of the global network and compares them with the reference image of the desired person, the authenticity of which is beyond doubt. An algorithm is used for processing the reference, which is based on the unique location of various parts of the face - the eyes, the nose and the mouth. It seems nothing new, but this software allows you to construct a huge base of Internet users. This technology is implemented in the form of widgets and can be embedded by the creators of web-portals on their sites. Search is conducted both on photos, and on video. This service has already been rated by companies like Flickr and YouTube. Supporters of technology believethat the development will simplify the search and identification. But there is a privacy policy that can not be violated.
Companies that present these innovations in their products, such as digital cameras and camcorders, do not back down. There are a lot of such companies, among them Ricoh, Fujifilm, Canon, Nikon® and others.

Many products of the companies are disassembled, there are advantages of software companies, as well as disadvantages. Based on this information, I made a decision to build a product compliance table with certain criteria, mainly for recognition and to find out the strengths and weaknesses of such emotion recognition modules:

findings

Although some companies claim that their product of reading face emotions is the most correct, it is important to note the fact that today there are technologies, but they only recognize features that are inherent and most appropriate for a given facial expression. But it is not always possible to say whether a person really feels this feeling, or is it just a trained smile, for example. If, however, to achieve a result so that the external expression matches the user's inner feelings, then, in my opinion, it is important to make sure that the participant is really comfortable and comfortable in expressing his feelings from the inside and showing them on his face and that he wants it. This is achieved using a full range of technologies, such as emotion recognition, facial expression recognition,speech recognition and intonation during the conversation of the person, the definition of semantic load and emotional experiences when writing and computer correspondence - all this greatly influences the final result. Also, you can use sensors.
As can be seen from the review, products are represented by many companies in different forms: from boxed software versions to web solutions based on the SaaS model (Software as a Service - software as a service) and applications in various types of equipment. Identified features and disadvantages.
In financial terms, the range of services varies greatly, but the vast majority of multifunctional solutions are expensive. Nevertheless, the demand for such systems is growing.