Christer Johansson

Statistics for Investigating Neuro-aesthetics: a case for Mixed Effects Models

We have investigated several interesting problems in neuro-aesthetics connected to experimental art history. One conundrum in art history is why the figure of Jesus has changed so much: from a curly haired Mediterranean man to the hippie-style Jesus we know today. A second conundrum is why the full front portrait was reserved for portraits of Jesus during most of the renaissance.
A general hypothesis in neuro-aesthetics is that nothing consistent is due to random chance but may be explained by how our perception is structured. Faces are extremely interesting because, as humans, we read so much information from dynamic facial expressions, such as face and gaze direction towards us or others, and we have clear aesthetic preferences for facial symmetry.
It is obvious that symmetry has some connection to health, which is very important for selecting a mate. However, for humans, it does not stop at mate selection. We may also judge trust by self-similarity. However, this may be in conflict with attractiveness that has a component of searching out beneficial genetic variance.
The whole field is complicated by culture, which may present us with a skewed sample of what is culturally attractive and the preferred looks of a trustworthy person. Artists have an interest in surviving from their art, so commissioned portraits will tend to be flattering. However, what is flattering may depend on the message the customer wants to send to the observer.

Paintings of Jesus took off during the Medieval to Renaissance as an image that was aimed for a mass appeal. The image would ideally attract new believers, and the commissioners certainly got feedback on the success of various styles and options. The portraits of Jesus were directed at a general believer and not necessarily a peer. This means that the image should be attractive to both men and women, from all walks of life. It seems to follow the logic that signals of divinity would also need reserved coding. We have suggested that face direction and gaze direction are two important dimensions.
Furthermore, we suggested androgyny, as represented in an image, may be universally more appealing. One reason is recognition of self, as both sexes may see something of self, as well as something of the opposite sex. However, androgyny in a male figure may signal high levels of both youth and health by heightened symmetry in the image.
Galton (1878) created composite images of criminals in order to find features that criminals had in common. He was surprised to find that the composite images became beautiful. The effect was that the random imperfections cancelled out, and the faces became more symmetrical.
It is also clear that the image of Jesus has adapted to the local communities, in interesting ways. In southern Europe, the image tends to have a darker skin tone and brown eyes. The further north in Europe, the paler the complexity and the bluer the eyes. Indeed, it looks like the images have adapted to the local communities in a non-random cline. 

Method

How to create controlled images
Figure 1 illustrates the procedure of pairwise morphing (averaging) images to form a prototypical portrait consisting of features from many pictures. The algorithm we used assumes pairwise averaging. This means that we are restricted to using a power of two images (2, 4, 8, 16, etc) to reach the final prototype with an equal contribution from all involved images. For the participant prototypes (female, human, male) we used 16, 32, and 16 images (cf. Figure 2). Adding the Jesus prototype results in a feminized, humanized, and masculinized images of Jesus.

Figure 1. Pairwise morphing resulting in a prototypical Jesus image
composed from four famous images of Christ.

Figure 2: Prototypes created from participants. All created by pairwise morphing.
Upper row: Female, Human and Male prototypes.
The lower row shows the effect of adding the Christ prototype.

Ethics considerations

In our experiments we have used photographs of actual people. In such a setting, there are many ethical considerations. When you work with adult participants, they are allowed to give informed consent. Informed consent means that the participants have been presented with a fair description of the intended research, its scope and the limitations of usage, and that they have understood how this relates to their participation and any effects this may have for their interests.

If you work with children, they cannot give consent but only assent. The consent depends on a responsible caregiver, if the child agrees to participate. Working with sensitive groups will always require more careful planning and protection of the privacy of the participants. Therefore, it is easier to work with participants that can give consent, while the privacy of your subjects still needs to be considered.

Research should also have a plan for how the data will be disseminated, and how it will be stored. It could be prudent to take steps to guard the anonymity of the participants. We decided to not use the original pictures, but rather use manipulated images that morphed portraits such that the images would contain features of the original pictures, thus creating an image that may vaguely resemble many people. The morphing technique creates an artificial image by “averaging” an individual image with other images. After the experiment, it could be verified if participants had identified themselves or not.  There are many ways that portraits may be created from a collection of pictures. Morphing, or averaging, is just one option. Recently there are machine learning algorithms that may create “deep fakes”, or portraits that look like real people but are created algorithmically. This option may prove very interesting for the generation of experimental stimuli, as it allows us to manipulate various properties of the portraits to create better controlled stimuli.  Deep fakes may also avoid some problems associated identifiable portraits and copyright issues.

Anonymity can be an issue. When we collect a database of images of real people, they might be possible to identify from the image. It is also a possible problem that the images might be used for other purposes than the purpose of the original data collection. This requires the consent of the included individuals, and a careful description of the usage of the images. Is it allowed to publish the images?

We want to control the images such that only the features we are interested in are in focus. This entails covering clothes with a neutral cloth, asking subjects to remove facial jewelry (ear rings, nose rings). Asking people to remove make-up, covering hairstyles or covering tattoos, may be practically difficult. Controlling the sample to avoid highly individual features may be beneficial in some settings, and in other settings it might be desirable.

In an experimental setting it is crucial to control the experimental stimuli in the experiment. Ideally, there should be one clear contrast between a baseline condition and the experimental condition. However, reality is that there are often many factors. Often it is beneficial to use binary factors to describe the stimulus. Examples are: direct gaze vs. averted gaze, full front vs. profile, prototype vs. individual. There are also factors that we cannot vary within the experiment, for example the gender of our subjects. For the same subject there is only one gender. The same could be true for the items in a study. The gender of an image could be thought to be either male or female, young or old, etc.

We decided to use photographs taken by a professional photographer. The sessions used a daylight setting at the same place and the same time of day (roughly mid-day). Clothes were covered to limit the influence of clothing style. In a larger study, random differences may contribute to the ecological validity of a study but in a smaller study outliers may skew the data sets considerably as each item, each individual is a larger part of a small sample compared to their part in a larger sample. 

Two main sources of variance in an experiment are the subjects and the items. When we regard them as sources of variance, we refer to them as random effects that can be used to explain variance in the statistical model.

How to set up an experiment

The presentation of stimulus items may be controlled by computer software (common alternatives are SuperLab (for mac), eprime (windows), psychopy (demands some programming skills in python)). For our simple experiment we created a hot-or-not contest with two pictures presented simultaneously side-by-side. The task is simply to for the participant to select one of the pictures. It could be the most attractive portrait. It could be the person they would trust the most. There are more variants, for example participants could be asked to judge who was most happy of the two, or who was more likely to be a librarian, a musician, a banker, the most introvert, the most extrovert, etc. It depends mainly on your research hypothesis.
Our research hypotheses were: 1) prototypes would be preferred and processed faster (more fluently) and 2) female images would be judged more attractive and processed more fluently (partly due to symmetry), and 3) our participants would react more favorably to images that were based on themselves.  Our initial task was to select the most attractive, but we are also investigating a selection for the most trustworthy.

Training effects, fatigue effects and how to handle the order of presentations.
People will need some test items to get used to the task. These training items are usually unrelated to the main experiment but will flow seamlessly into the experiment as a warm-up for the participants. The subjects will gain experience with the experiment as it progresses.  Typically, they get slightly faster in making decisions up to a point when they may become fatigued and start to slow down or get more inaccurate. Therefore, it is vital to limit the length of the experiment and allow for pauses after about 10 to 15 minutes.
It is also very important that order of presentation is varied such that there is no association between when a stimulus was presented and the class of stimulus. This may be achieved using randomization, such that each participant gets a new random order of presentation.
It is important to vary the presentation on the screen such that there is no association between left or right image and the class of the image. In our experiment, the class of the image was either a prototype or an individual personalized image. This was presented exhaustively for all combinations, with both left and right presentations for all stimulus. Each subject had their own unique randomized sequence of events. We therefore expect that any effects due to the presentation order will cancel out over the entire experiment. Variance is estimated for both subjects and items simultaneously.

Hot-or-not contest

The task for the participants were set up as a hot-or-not contest where people would pick one of the pictures as more attractive than the other. Figure 3a and 3b presents two test events.

Figure 3a. Two pictures. One is a feminized prototype and one is the prototype with
an individual woman. Which one would you pick?

Figure 3b. two pictures. One is a masculinized prototype and one is the prototype
with an individual man. Which one would you pick?

The choice for each combination was recorded with the time it took to take the decision.
Unknown to the participants, it was always a choice between a prototype and an individualized prototype. Some of the individualized presentations involved the participant self.
If you picked the image on the right hand side in Figure 3a and the image on the left hand side in Figure 3b you would have picked the feminized and masculinized prototypes.

Figure 3c. The winners.

How to evaluate

When we build a statistical model of our data the first question is which of the variables is the variable we want to explain. This variable is called the dependent variable. The variables we want to use as our fixed factors to explain the dependent variable are sometimes called independent variables. The variables that we regard as the (main) sources of variance are the random effects. Typically the random effects stem from the subjects and the test items. The random effect can be further divided up by “slopes”, that is the fixed factors that we think may be different for subjects or items. In order to use a slope we need at least two measurements per subject or item for the levels of each fixed factor we include as a slope.

Let us look at an example. The hot-or-not contest could be described by the following formula if we are interested in explaining the reaction times (rt) in terms of class of the image (personalized or prototype), gender of the image (male or female), p-gender of the participant, self in the image, and potentially interactions between these factors. If we include all interactions we would need too much data to estimate all the effects, and we would end up with a model that is too hard to interpret. We may choose to do the testing first overall for both male and female participants and then divide the dataset up to see if male and female participants make different decisions. We analyze the reaction times for the chosen image, and it is always a competition between a prototype and a personalized prototype.

1. rt~class*gender+self +((self*gender*class) | participant) + (1 | item)

In (1) we are investigating if reaction time can be explained by (marked by ~) class and gender of the image and the reaction to self in the image. We assume that the participants may react differently to self (or not) in the image as well as to gender and class and their interaction (marked by *). Each image does not vary in the presentations, it is always what it is (prototype/person, male/female/jesus) so each image is only considered to have different variance due to their content). In a linear regression model a starting point is called an intercept, and is marked by 1 in the formula. All slopes have a starting point by definition.
If rt turns out to have a long tail, as is most often the case, we will analyze rt as its transform using the (natural) logarithm of reaction times. Logarithms will compress the scale, with the effect that the distribution will be closer to the desired normal distribution. We may transform back to reaction times when we present the results, and we will have asymmetric confidence intervals due to the back transformation. The model to use is a Linear Mixed Effects model with Random Effects. This model is implemented in R by the function lmer.

However, it also possible to consider the choice as the variable to be explained. Formula 2 expresses this as statistical model. For the moment, we will ignore the impact of reaction time (rt).

1. class~gender+self + (self*gender|participant) + (1 | item)

We are now in a situation, where we have a categorial variable to explain, and we must accommodate for this in the model we choose. We cannot use a model based on the normal distribution, but we may use a model based on the binomial distribution. In this case, we assume two “boxes” (prototype and not prototype) and a logistic transfer function. This will take us into a class of generalized linear models that allows linear logistic regression with random effects. The model is realized in R using glmer, and the results can be presented using a graph of the odds ratio, as well as the estimates of significance. (MORE WORK REMAINS – need to redo the analysis.)

However, there is a simpler model that can be tried out for choice. We may look at associations between variables using an association plot. This plot makes use of the chi-square calculations, which compare observed and expected frequencies, and it is effective when you can express your data as a two-dimensional table. The weak point of this analysis is that it does not fully exploit the information about participants and items. However, it is easier to understand and explain, in terms of associations. The plot allows us to look into which cell combinations in the table, and see which combinations are significantly higher or lower than expected. This is implemented in R by the assoc function (from the vcd package).
Figure 4 shows the result when we combine the choice and the type of prototype (this has more options, since we restricted ourselves to a two-dimensional table). Table 1 shows the underlying frequencies.

Figure 4: Prototypes are: PH (Human), PHX (Human with Christ), PJ (Jesus Christ), PM (Man), PMX (Man with Christ), PW (Woman), PWX (Woman with
Christ). Differences are significant. Red marks cells with lower than expected freq­uencies, blue are higher than
expected.  χ²₍₆₎=154.4, p<0.001, Φ_c=0.096

Table 1: Frequency of choice for person (PE) or prototype (PR), competition for each prototype.

The association table is an option that quickly gives us a visualization of the data. However, it is difficult, or impossible, to analyze the intricate interactions and the sources of variance. Often, we would have to generate many association tables to investigate our data. The good point of association tables is that they quickly show where there are are associations in our data. In the data above, it is quite clear that prototypes for Jesus and Woman are chosen much more than would be expected by random chance. This would lead us to speculate about a possible link between femininity and the Jesus images, which was previously shown in a different experiment.

Conclusion

We have illustrated some common methods for evaluating experiments by statistical models. Currently Mixed Effects models are very popular. This type of model allows the explicit modeling of uncertainties in the data that may stem form participants and items. For continuous dependent variables (such as reaction time) we may use the lmer-model. This assumes that we either have normally distributed data, or that we can achieve normal distribution by for example a logarithmic transform. The logarithm function is suitable because it is a monotone function that preserves the order of our data points while compressing the scale so that potential outliers will not have too much impact on our results. Alternative approaches might be to use non-parametric tests such as Friedman tests. The drawback of non-parametric tests is that they provide less information and may not use the information in our datasets optimally.
For categorial choice data we have indicated the use of a linear logistic regression model that may optimally consider our sources of uncertainty in the data. One option is to explore generalized linear models in R and make sure to choose a sensible distribution function and an appropriate linking function. These models are getting increasingly popular, as they both handle the uncertainties in the data in a reasonable manner without the need for extensive repeated tests and they generate impressive graphs that are fairly easy to interpret and present. We have also shown the use of association tables. Even though they do not handle uncertainties in the data as elegantly as the other models they are robust and fairly easy to interpret.

We have not gone into the field where you have many dependent variables, often sampled simultaneously from the same participants. This is often the case in for example Brain Imaging techniques such as fMRI and ERP. Such techniques demand more experience with the field, and they are often implemented by team of experts using specialized software. Obviously, this may be a disadvantage for small research groups.

Lab on gender and prototypes

Cards are printed out to make up a deck of cards consisting of 5 feminized prototypes, 5 masculinized prototypes, one Jesus prototype, one male prototype and one female prototype. The reason for using personalized prototypes is to make the task less obvious.

You will need about 10 subjects to perform the task.

The task is either to sort on attractiveness or trustworthiness, from left to right.
Assume that we ask to sort on attractiveness, most attractive first.

Each card is marked on the backside for male or female, or which type of prototype.

The sequences can be read and marked as in the example that follows.

m jp m fp f  f mp f  m f   m   f     m
0  1   2  3  4 5  6   7  8  9  10  11  12

For each card we count the number of cards to the left.

M         F          JP         FP        MP
0          4          1          3          6
2          5
8          7
10        9
12        11

We may test the sorting orders between male and female with a paired Wilcoxon test.
In a second test we may test if everybody have the same ratings, and if the gender of the subjects influences the results.

We may gather the ratings for each of the prototypes for each subject and use an unpaired (independent measures) Wilcoxon test to see if there are differences between the prototypes.

There are many things to explore. For example, will people how rate males higher also prefer the male prototype?