Words: 4572
Perception, 2003, volume 32, pages 285 ^ 293 DOI:10. 1068/p5027 The role of eyebrows in face recognition Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 45 Carleton Street, E25-201, Cambridge, MA 02142, USA; e-mail: sadr@mit. edu; sinha@ai. mit. edu Received 23 April 2002, in revised form 4 November 2002; published online 7 March 2003 Javid Sadro, Izzat Jarudi, Pawan Sinhao Abstract. A fundamental challenge in face recognition lies in determining which facial characteristics are important in the identification of faces.
Several studies have indicated the significance of certain facial features in this regard, particularly internal ones such as the eyes and mouth. Surprisingly, however, one rather prominent facial feature has received little attention in this domain: the eyebrows. Past work has examined the role of eyebrows in emotional expression and nonverbal communication, as well as in facial aesthetics and sexual dimorphism. However, it has not been made clear whether the eyebrows play an important role in the identification of faces.
Here, we report experimental results which suggest that for face recognition the eyebrows may be at least as influential as the eyes. Specifically, we find that the absence of eyebrows in familiar faces leads to a very large and significant disruption in recognition performance. In fact, a significantly greater decrement in face recognition is observed in the absence of eyebrows than in the absence of eyes. These results may have important implications for our understanding of the mechanisms of face recognition in humans as well as for the development of artificial face-recognition systems. Introduction Evolutionary history has seen a considerable reduction in the amount of hair on the human face (McNeill 2000; figure 1). As such, the presence of the eyebrow might seem a curiosity. Do the eyebrows in fact serve a useful purpose or are they merely an evolutionary vestige? While it is perhaps true that eyebrows may provide the eyes modest protection against such things as rain and perspiration, it is perhaps more relevant that the eyebrows also appear to serve a number of functions that are more visual in nature.
Eyebrows may serve as high-contrast lines that give the appearance of the brow greater clarity and emphasis, and their associated musculature allows for sophisticated, often involuntary gestures that may be discerned from a relatively large distance. As such, the eyebrows appear to play an important role in the expression of emotions and in the production of other social signals, and they may also contribute to the sexual dimorphism Figure 1. Overall, humans have relatively little facial hair as compared to other primates; the conspicuous presence of the hair forming the eyebrows is somewhat intriguing.
A number of selection pressures may be responsible for the development and persistence of the eyebrows over the course of primate and especially human evolution. A color version of this figure can be seen at http://www. perceptionweb. com/misc/p5027/. oAuthors to whom correspondence and requests for materials should be addressed. 286 J Sadr, I Jarudi, P Sinha (ie sexual differentiation) and even the aesthetics of faces. Here, we begin by reviewing past work concerned with the appearance and functions of the eyebrows, then proceed to empirically investigate whether the eyebrows also play an important role in the recognition of faces.
Practitioners in the field of facial aesthetics, such as make-up artists and cosmetic surgeons, have long appreciated the influence of eyebrows on attractiveness (eg Cosio and Robins 2000). During the 18th century, in fact, in Western Europe full eyebrows were considered so essential to facial beauty that some upper-class women and courtiers would affix mouse hide to their foreheads. The perceived importance of the eyebrows for enhancing beauty has not waned to this day.
Currently, it is relatively common cosmetic practice to use tweezers or depilatories to narrow and accentuate the arch of the eyebrows, as well as to remove any hair at the bridge of the nose. Cosmetics may also be used to alter the color (especially the darkness) and exaggerate the shape and length of the eyebrows. Moreover, several cosmetic surgery procedures, including botulinum toxin (`botox’) injection, permanent tattooing, and surgical tucks and `lifts’, specifically target the appearance of the eyebrows. One may also note the sexual dimorphism in primate eyebrows.
Given that the brow ridge may have been an important, sexually distinctive characteristic of our early ancestors’ faces, it is not surprising that recent studies have found an important role for eyebrow thickness in discriminating between male and female faces (Bruce et al 1993). A relative thinness of the eyebrows, and perhaps to some extent the separation of the eyebrows (and the underlying bony ridge) into two disconnected arcs, might serve as part of a quick diagnostic of juveniles, females, and fine-featured males.
Indeed, one could speculate that this underlies, at least in part, the aesthetics driving the image of the well-groomed eyebrow in modern cosmetic practice. In the other extreme, the thick swatch of hair across the brow of some older human males can give the underlying structure a lowered, protruding appearance somewhat reminiscent of the exaggerated brow ridge seen, for example, in mature male gorillas. In the latter, of course, this could be seen as quite an intimidating feature, especially by less mature competing males (not to mention by a human observer who might have mistaken these herbivores for aggressive predators).
Another domain in which the importance of eyebrows has been acknowledged is that of emotional expression, not to mention nonverbal communication in general. Unlike the fixed, bony brow ridge that may have graced our ancestors, the `false’ brow of eyebrow hair is mobile enough to produce a broad range of facial signals. Our eyebrows can communicate the extremes of aggression or fear, and, in combination with other coordinated facial movements, the entire range of human emotions. Ekman and colleagues (eg Ekman and Friesen 1978; Ekman 1979) have systematically examined the function of eyebrows in facial expressions that signal emotions.
In developing their system of facial-expression analysis, Ekman and Friesen (1976, 1978) compartmentalized the face into three regions: an upper component consisting of the eyebrows and forehead; a middle component consisting of the eyes and cheekbones; and a lower component consisting of the nose, mouth, and chin. These researchers extensively investigated these three components in facial expressions of the six emotions posited to be universally expressed and recognized across cultures (happiness, sadness, anger, fear, disgust, and surprise; eg Ekman and Friesen 1971; Ekman 1993).
Expression of each of the six emotions was seen to involve a distinct combination of movements in the three facial components; and the upper component, notably the eyebrow, was found to play a key role in the expression of a number of emotions, including happiness, surprise, and anger. For example, in the expression of anger, the eyebrows are typically pulled downwards and inwards, perhaps in combination with squinting eyes and a tightly closed mouth. It is not surprising that cartoonists recognize and exploit the
Eyebrows and face recognition 287 Figure 2. These cartoon faces are identical except for their eyebrows. Alone, or in concert with other facial movements, changes in the angle, height, and curvature of the eyebrows can drastically alter the emotional expression of a face and may play an integral role in nonverbal communication. power of the eyebrows to convey emotions in even the simplest of line drawings. An illustration of this is depicted in figure 2, in which the three faces are identical except for the angle and height of their eyebrows.
It can be seen that the movement of the eyebrows alone may be enough to greatly alter the emotional expression of a face. Closely related to the contribution of eyebrows to the facial expression of emotions is their involvement in other forms of communication. In addition to past work examining the functional role of eyebrow movements as conversational signals (eg Ekman 1979), it is also interesting to observe the great extent to which the eyebrows are involved in purely nonverbal communication.
An excellent demonstration of this comes from the domain of sign language (eg American Sign Language; Baker-Shenk 1985). Here, in the complete absence of vocal prosody, facial gestures (and eyebrow movements in particular) serve to modulate and complement that which has been signed by the hand and body. For example, raising the eyebrows quite naturally serves to recast a hand-signed expression as a question rather than a declarative statement, much in the same way that rising pitch can signal a question in a vocal utterance.
Nevertheless, in spite of the existing literature that has explored the involvement of eyebrows in such domains as facial aesthetics, gender discrimination, emotional expression, and nonverbal communication, their role in the recognition of faces per se has gone largely unexamined. Prior studies exploring the significance of different facial features for the task of recognition have typically proposed a hierarchy with the eyes as the most important features, followed by the mouth and the nose (Davies et al 1977; Haig 1986; Fraser et al 1990).
The role of the eyebrows per se remains unclear. Intuitively, it would seem that, given the diversity of the appearance of eyebrows (figure 3) and of their configuration relative to the hairline and the eyes, the visual system could benefit from including the eyebrows in its representation of faces. Yet, the authors of only a few face-recognition studies examining the relative importance of different facial features have included the eyebrow in their experiments (eg Haig 1986; Maruyama et al 1988).
For example, Haig (1986) treated the eye and eyebrow as a unit while interchanging features among target faces and found that, next to the head outline, subjects found the eye/eyebrow feature substitution most influential in making a modified face similar to an original `target’ face. Most investigations of such feature salience, however, have simply omitted the eyebrow for their testing, perhaps because the eye has still been shown to be the most significant internal recognition feature (Davies et al 1977; Shepherd et al 1981; Fraser et al 1990).
One shortcoming of these studies is their use of unfamiliar or novel facial stimuli. For such images, face recognition has been shown to depend more on external features (eg head outline) than internal features, such as the eyes or eyebrows (Young et al 1985; 288 J Sadr, I Jarudi, P Sinha Figure 3. Considering the great diversity in the appearance of eyebrows across individuals, one may expect the eyebrows to constitute an informative attribute for the task of face recognition.
Our experiment was designed to assess whether the human visual system does in fact rely on the appearance of eyebrows in order to identify people. A color version of this figure can be seen at http://www. perceptionweb. com/misc/p5027/. Bruce et al 1999). In contrast, familiar faces appear to be better identified from internal features and may be more appropriate stimuli for assessing the role of eyebrows and other features in face recognition (Ellis et al 1979; Young et al 1985). As such, the stimuli chosen for the current study were images of famous faces.
Further, we sought not only to investigate the involvement of eyebrows in the recognition of these famous faces, but also to compare any such contribution to that of their most favored neighbors, the eyes. The goal of our experiment was to address this issue directly, by determining whether and to what extent the absence of eyebrows might reduce the ability of individuals to recognize otherwise familiar faces, and by quantitatively relating any such effect to the corresponding detrimental effect that might result from the absence of eyes.
Of course, it may be worth noting that memory-based face recognition, the focus of the current study, might rely on processes different from those involved in, for example, match-to-sample comparisons or subjective judgments of facial appearance of similarity, even for the same set of stimuli. 2 Methods 2. 1 Participants Eighteen subjects (thirteen male, five female; mean age ? 23. 2 years) participated in the experiment. The volunteers were members of the MIT community, and the majority were students. 2. 2 Stimuli Fifty faces of famous Caucasian men and women, twenty-five of each sex, were collected as reference images for the experiment.
The majority of the celebrities were television or movie stars, but the stimulus set also included images of some famous musical performers, politicians, and fashion models. From this set of fifty images, we generated two new sets based on the feature omission technique used in earlier studies to judge the importance of a feature to face recognition (eg Fraser et al 1990). Using Adobe Photoshop, we created one set of face images without eyebrows and another set of images without eyes. The eyes and eyebrows were removed with the `clone’ tool in Photoshop; in this way, the region of the eyes or eyebrows was replaced with the urrounding skin texture/color while generally preserving the bone structure of the eye socket or brow (figure 4). To be sure, there may be a certain subjective, perhaps even artistic, aspect to such a procedure, as well as in the assessment of the stimuli thus produced. That said, the images were repeatedly scrutinized, both during and following eyebrow or eye erasure, to ensure that the manipulations entailed only as much alteration as necessary to remove the feature of interest (eg eyebrow hair) while restricting Eyebrows and face recognition 89 Figure 4. Sample stimuli from our experiment assessing the contribution of eyebrows to face recognition: original images of President Richard M Nixon and actor Winona Ryder, along with modified versions lacking either eyebrows or eyes. the introduction of any concomitant artifacts (eg any changes in the color, shading, or shape of the brow ridge inconsistent with the original image). When such an artifact could be observed in a stimulus, whether at the normal scale or higher magnification, the relevant portion of the image was redone.
The original photographs of the famous faces used to make all the test images (and used unaltered as the reference images in the second half of the experimentosee below) had first been cropped to retain only the central part of the face. This was done to avoid ceiling effects in the subjects’ recognition performance due to potential reliance on other, especially external, facial features (hair, ears, full head outline, etc, not to mention other extraneous cues such as clothes). The images were presented on a color CRT monitor at a viewing distance of approximately 60 cm. Each image subtended, on average, 3. deg64 deg of visual angle and was displayed at the center of the screen on a gray background. The experiment was conducted in a room with low ambient illumination. 2. 3 Procedure In the first half of the experiment, subjects viewed, in random order, twenty-five celebrity images without eyebrows and twenty-five without eyes. Any celebrity depicted in one condition did not also appear in the other, and the assignment of celebrities to the no-eyebrows or the no-eyes condition was randomized across subjects. In the second half of the experiment, subjects viewed all fifty reference images, again in random order.
These original, unaltered images were used to assess (and thereby control for the variability in) subjects’ familiarity with each celebrity, and performance on the reference images was subsequently used to normalize the subjects’ performance with the altered stimuli. For each of the hundred images presented, subjects were asked to 290 J Sadr, I Jarudi, P Sinha press a keyboard or mouse button as soon as they recognized the celebrity. After such a response, the celebrity image was replaced with a text input area; subjects were asked to identify the person in the previous image by typing either their name or some unique identifying information. Such an identifier could simply specify a job or action for which the celebrity may have been famous; for example, for actors this would include the name of a movie, television show, or character with which he or she may have been associated. ) There was no time limit for the subjects to input their response. The experimenter’s instructions described the general design of the experiment and explained the subjects’ task to be the identification of celebrity images, some of which would lack eyes or eyebrows.
Prior to beginning the experiment, subjects performed five practice trials (with images not used in the subsequent experimental trials) in order to familiarize themselves with the experimental procedure. 3 Results Figure 5 shows the results averaged across eighteen subjects, with performance reported as the fraction of correctly identified celebrities relative to the control condition with unaltered face images. (That is, each subject’s performance in the no-eyes and no-eyebrows conditions was assessed only in terms of his or her ability to recognize the same celebrities when shown their unaltered images in the second half of the experiment.
Subjects were able to identify, on average, 30. 1 of these fifty original images, SE ? 2X6. ) Consistent with previous reports, the absence of eyes significantly reduced subjects’ ability to recognize the celebrities (mean ? 55X8%, SE ? 3X2%; p 5 0X001, paired t-test). In addition, however, the data reveal two other interesting results. First, with respect to the control condition, the celebrity images lacking eyebrows also proved remarkably difficult for subjects to identify (mean ? 46X3%, SE ? 4X4%; p 5 0X001).
Second, recognition performance for faces without eyebrows was, in fact, significantly worse than that for faces without eyes (mean difference ? 9. 5%, SE ? 4X8%; p 5 0X05). Surprisingly, therefore, face recognition appears to be disrupted even more by the absence of eyebrows than by the absence of eyes. 60 Recognitiona% 40 20 0 No eyebrows No eyes Figure 5. Averaged data from our experiment assessing the contribution of eyes and eyebrows to the identification of faces. Consistent with previous findings, face recognition was impaired for images lacking eyes.
Surprisingly, however, the absence of eyebrows caused an even greater decline in the subjects’ ability to recognize familiar faces. 4 Discussion Our results suggest that eyebrows play an important role in human face recognition. This finding is surprising given that past investigations of face recognition, not to mention common intuitions asserting the distinctiveness and expressiveness of the eyes, have largely ignored the eyebrows. In order to account for these results, we consider a number of possibilities. Eyebrows and face recognition 291
Perhaps the simplest explanation of these data could be that the images lacking eyebrows were merely more different from the original images than were the ones lacking eyes. That is, at the basic image level, there may have been a greater difference between the original and no-eyebrows images than between the original and no-eyes images. To explore this possibility, we calculated both the absolute and the squared difference (ie the L1 and L2 distances; Duda et al 2001) between the original images and the corresponding images in the no-eyebrows and the no-eyes conditions.
Contrary to the above prediction, we found that the differences between the original and no-eyebrow images were significantly less than the corresponding differences between the original and no-eyes images [ p 5 0X05 for the absolute differences (L1); p 5 0X01 for the squared differences (L2); figure 6]. Thus, it does not seem that our data can be readily explained in terms of simple low-level image differences. 18 Absolute difference (L1) from original image610A4 16 14 12 10 8 6 4 2 0 No eyebrows No eyes 10 9 8 7 6 5 4 3 2 1 0 Squared difference (L2) from original image610A6
No eyebrows No eyes Figure 6. Average absolute and squared differences (L1 and L2 distances) between the original celebrity images and those lacking eyebrows or eyes. Whereas the results of our experiment indicate faces lacking eyebrows were significantly more difficult to recognize than those lacking eyes, the stimulus images lacking eyes were in fact significantly more different, at a basic image level, from the original images. Another possibility to consider is whether the difficulty in recognizing the modified images may be due to the unnatural appearance of a face without yes or eyebrows. However, as has been found in previous studies in which the feature omission technique was used, unnaturalness alone cannot account for decrements in recognition performance. Recognition performance is found to correlate with the perceptual salience of the facial feature rather than being governed by whether a face looks unusual. Thus, for instance, omission of the nose leads to an unnatural face without significantly compromising its identifiability (Fraser et al 1990).
More to the point, however, in the context of our study, there is little reason to assume that faces lacking eyebrows are any more unnatural than those without eyes; indeed, in the real world, a face without eyes would likely be perceived as far more odd than one without (or with extremely fair) eyebrows. And yet, as our data suggest, the oddity of eyeless faces does not lead them to be recognized more poorly than, or even as poorly as, those without eyebrows. Ultimately, we are led to conclude that eyebrows do indeed play an inherently important role in face recognition.
If this is the case, however, how might one reasonably explain the perceptual significance of eyebrows in face recognition? There are several possibilities. First, as discussed above, eyebrows appear to be very important for conveying emotions and other nonverbal signals. Since the visual system may already be biased to attend to the eyebrows in order to detect and interpret such signals, it may be that this bias also extends to the task of facial identification. Second, for a number of reasons, eyebrows may serve as a very `stable’ facial feature.
Because they tend to be relatively high-contrast and large facial features, eyebrows can survive substantial image degradations. For instance, when faces are viewed at a distance (or, similarly, at low 292 J Sadr, I Jarudi, P Sinha image resolutions), the eyebrows continue to make an important contribution to the geometric and photometric structure of the observed image. Also, since eyebrows sit atop a convexity (the brow ridge separating the forehead and orbit), as compared to some other parts of the face, they may be less susceptible to shadow and illumination changes.
Finally, while for a given face the eyebrows may be considered a robust and consistent facial feature in these respects, the eyebrows are equally attractive as a salient and informative characteristic for facial identification owing to their great diversity of appearance across different faces (figure 3). As with other facial features, the eyebrows may exhibit changes in appearance over the time scale of many years, but over weeks or months these attributes of the eyebrows can serve as reliable cues to identity.
While the current findings identify an intriguing feature of human face recognition, they also prompt new questions well met by further experimentation. For example, beyond the mere presence or absence of the eyebrows, one could next study the effects of manipulating the appearance of the eyebrows along various dimensions (shape, color, location, etc), or of replacing them altogether with those from other faces.
Producing such stimuli by manipulating existing images of celebrities or by creating entirely new stimuli with the use of cosmetics and models (in the extreme, the eyebrows of a willing volunteer can be redrawn entirely after being physically `erased’ with cosmetic wax, sealer, and foundation; Aucoin 1997) poses certain practical challenges, but the results of experiments using such stimuli would be of great interest.
In conclusion, as discussed above, one may speculate that the eyebrows have evolved (and persisted) owing to selection pressures associated, at least in part, with their role in facial attractiveness (Cosio and Robins 2000) and sexual dimorphism (Bruce et al 1993)oand, thus, in mate selection and sexual competitionoas well as in nonverbal communication and emotional expression (eg Ekman 1979). In fact, research in the latter field has shown the eyebrows to be, in a kinematic sense, the most expressive part of the face (Linstrom et al 2000) and, we would suggest, a facial feature whose gestures would be easily recognized at a distance.
The findings of our experiment fit well in this context and suggest that the salience of the eyebrows extends into the domain of face recognition, a domain where their study could shed further light on the representations and mechanisms that enable us to recognize one another. In turn, an improved understanding of the human visual system may contribute to the development of artificial systems that better mimic its often astonishing abilities. Acknowledgments. The authors thank Aude Oliva, Antonio Torralba, Gadi Geiger, Brad Duchaine, Sayan Mukherjee, and Tabitha Spagnolo Sadr. This research was supported by a grant rom the DARPA Human ID program and an Alfred P Sloan Fellowship in neuroscience to Pawan Sinha, and the John Reed and Meryl and Stewart Robertson funds for undergraduate research at MIT to Izzat Jarudi. Javid Sadr is a Howard Hughes Medical Institute predoctoral fellow. References Aucoin K, 1997 Making Faces (Boston, MA: Little, Brown and Co) Baker-Shenk C, 1985 “The facial behavior of deaf signers: Evidence of a complex language” American Annals of the Deaf 130 297 ^ 304 Bruce V, Burton A M, Hanna E, Healey P, Mason O, Coombes A, Fright R, Linney A, 1993 “Sex discrimination: how do we tell the difference between male and female faces? ‘ Perception 22 131 ^ 152 Bruce V, Henderson Z, Greenwood K, Hancock P, Burton A M, Miller P, 1999 “Verification of face identities from images captured on video” Journal of Experimental Psychology: Applied 5 339 ^ 360 Cosio R, Robins C, 2000 The Eyebrow (New York: Harper Collins) Davies G, Ellis H, Shepherd J, 1977 “Cue saliency in faces as assessed by the `Photofit’ technique” Perception 6 263 ^ 269 Duda R O, Hart P E, Stork D G, 2001 Pattern Classification 2nd edition (New York: John Wiley) Eyebrows and face recognition 293
Ekman P, 1979 “About brows: Emotional and conversational signals”, in Human Ethology: Claims and Limits of a New Discipline Eds M Von Cranach, K Foppa, W Lepenies, D Ploog (Cambridge: Cambridge University Press) pp 169 ^ 202 Ekman P, 1993 “Facial expression of emotion” American Psychologist 48 384 ^ 392 Ekman P, Friesen W V, 1971 “Constants across cultures in the face and emotion” Journal of Personality and Social Psychology 17 124 ^ 129 Ekman P, Friesen W V, 1976 “Measuring facial movement” Environmental Psychology and Nonverbal Behavior 1 56 ^ 75 Ekman P, Friesen W V, 1978 Facial Action Coding System (Palo Alto, CA: Consulting Psychologists Press) Ellis H, Shepherd J, Davies G, 1979 “Identification of familiar and unfamiliar faces from internal and external features: some implications for theories of face recognition” Perception 8 431 ^ 439 Fraser I H, Craig G L, Parker D M, 1990 “Reaction time measures of feature saliency in schematic faces” Perception 19 661 ^ 673 Haig N D, 1986 “Exploring recognition with interchanged facial eatures” Perception 15 235 ^ 247 Linstrom C J, Silverman C A, Susman W M, 2000 “Facial-motion analysis with a video and computer system: a preliminary report” American Journal of Otology 21 123 ^ 129 Maruyama K, Masame K, Endo M, 1988 “Sameness or not-sameness of the corresponding features and their summation effect in similarity judgment of faces” Tohoku Psychologica Folia 47(1 ^ 4) 74 ^ 84 McNeill D, 2000 The Face: A Natural History (Boston, MA: Little, Brown and Co) Shepherd J, Davies G, Ellis H, 1981 “Studies of cue saliency”, in Perceiving and Remembering Faces Eds G Davies, E Hadyn, J Shepherd (London: Academic Press) pp 105 ^ 132 Young A W, Hay D C, McWeeny K H, Flude B M, Ellis A W, 1985 “Matching familiar and unfamiliar faces on internal and external features” Perception 14 737 ^ 746 ? 2003 a Pion publication ISSN 0301-0066 (print) ISSN 1468-4233 (electronic) www. perceptionweb. com Conditions of use. This article may be downloaded from the Perception website for personal research by members of subscribing organisations. Authors are entitled to distribute their own article (in printed form or by e-mail) to up to 50 people. This PDF may not be placed on any website (or other online distribution system) without permission of the publisher.