Do Reading Tests Paint A False Picture?

For some time now researchers have witnessed a trend for girls to be ahead of boys in reading tests. This trend is consistent across all age groups and across most countries. However mysteriously by the time young adulthood dawns, the difference in reading skills has evaporated. Is this an as yet unexplained aspect of nature or is it attributable to the design of the tests themselves?

International reading studies like PIRLS and PISA indicate girls are better at reading than boys. The gap is apparent amongst 10-year olds measured in PIRLS, and even wider in the 15-year olds who participate in PISA across the OECD.

As currently designed, the reading tests assess whether pupils can; assimilate information from the text, draw simple conclusions, interpret and compare information, and assess language, content and literary devices contained in the text. Girls consistently perform better than boys regardless of which aspect is being evaluated. That marked difference disappears by adulthood.

When the reading skills of 16-24-year-olds are tested, the once stark gender differences have shrunk or have disappeared entirely. The PIAAC study, which tests adults’ literacy, numeracy and ICT skill demonstrates this clearly.

Despite reading being an important factor in participating in education, work, and society, there are no major gender differences in the Nordic region in these areas. While more women than men have higher secondary or tertiary qualifications, women are behind men in employment rates, participation in society and income.

Norwegian men are more likely to hold managerial positions than women, they are more involved in local politics, and Norwegian men earn more than Norwegian women.

In explaining the difference between girls and boys reading skills, several hypotheses have been explored. Intelligence has been rejected since girls do not have higher IQ than boys. Teaching methods are also unlikely to be the culprit. Some researchers proposed that expectations are different for girls than boys. However, it does not fully explain the differences. An answer as to why this difference disappears when pupils leave secondary school is still proving elusive.

Literacy researchers Oddny Judith Solheim and Kjersti Lundetræ of the Norwegian Reading Centre, University of Stavanger explored whether the test design could be the culprit for why differences that become more pronounced as children progress through school, then disappear in 16-24-year-olds. Researchers compared the way the PIRLS (5th grade), PISA (10th grade) and PIAAC (adults) tests are designed, their implementation and their test measurement methodology

All the tests use the same definition of ‘reading literacy’: It is about being able to understand and use written text, and PISA and PIAAC also assess the ability to reflect on and evaluate texts.

“Based on earlier research, it appears that PIRLS and PISA, i.e. the tests used in schools, are designed in a way that may favour girls. PIAAC is designed differently. This could be one explanation as to why we are seeing gender differences in the results,” observed Solheim.

Both the PISA and PIRLS tests contain numerous ‘continuous texts’, containing long texts, which may be descriptive, narrative, or explanatory. Previous research indicated girls and women are better at reading ‘continuous texts’ than boys and men. Boys and men are better at reading ‘non-continuous texts’, such as graphs, forms, and advertisements. Several studies also show the differences are greater in favour of girls when pupils have to read fiction, than when they read factual texts.

In the PIRLS tests for 10-year-olds, the majority of texts are continuous, and the distribution of fictional texts and factual texts is the same. In PISA, 60 percent of texts are continuous, and 15 percent are fictional. However, in PIAAC, there are equal numbers of continuous texts and non-continuous texts, with the participants only having to read factual texts.

“Since we know that it is an advantage for girls to read long, fictional texts, it could be giving them an advantage to provide them with this type of text in the reading tests, which could affect the results in terms of measuring pupils’ skills,” says Solheim.

A number of international studies show girls and women are better at writing than boys and men, although there are no differences between young adult men and women when their reading skills are measured.

Some PISA and PIRLS questions are multiple-choice. However, in recent years, more of the questions have been open-ended, requiring the pupils to provide a written answer. This approach is believed to provide a clearer picture of the pupils’ understanding. Reading is then being measured through writing, which gives girls an advantage. In PISA, where the difference between girls and boys is greatest, 65 percent of the exercises involve writing. In PIAAC, on the other hand, participants do not need to write, and instead select words, sentences or extracts from the texts.

Again, several studies show gender differences are greater in written exercises than in multiple-choice questions, and boys have a greater tendency to skip the written questions. For this reason, the two literacy researchers believe that PISA and PIRLS are more girl-friendly than PIAAC.

Motivation is an important aspect when we consider how reading skills are measured. Differences in motivation to do the tests could explain a great deal of why girls appear to read better than boys at school, but not when they reach adulthood.

For example, earlier research shows it is more difficult to motivate boys to be interested in a text than girls. The gender of the protagonist, the subject of the text and attitudes to the text or general subject play more of a role for boys in how well they perform when they have to read than for girls. For this reason, the researchers at the Norwegian Reading Centre believe test designers should take into account boys’ motivation to read the texts they are given in the tests.

We also know that girls are more likely to comply with expectations than boys. Boys are more likely to question the point of a test. The biggest differences in the tests are in the 10th grade, a time when pupils are facing many other challenges from their school.

“Since we know that boys are more critical about doing things that have no direct significance for them, it is conceivable that they are more likely to avoid expending energy on a test that will not affect their qualifications. Motivation could also explain part of the reason why the differences are greater at lower secondary school than primary school, since it is well known that teenagers are more likely to question authority, such as the school, than younger children,” says Solheim.

While the pupils performed the tests at school, the adult participants in PIAAC were invited to do the test in their own home, supervised by a PIAAC representative. Participants in PIAAC were also rewarded once they had completed the test. The researchers suggest it is reasonable to assume that you might feel more obligated to do your best, sitting a test at home with a supervisor than someone sitting in a classroom who will not receive any particular reward. It is conceivable that the boys and men who underwent PIAAC were more likely to show the full extent of their reading ability, than the boys who took part in PISA.

The difference in reading between girls and boys has been highlighted as an educational challenge in most OECD countries, including Norway. Lundetræ and Solheim believe their findings should be incorporated into the design of reading skills tests, and in how they are interpreted.

“Reading is described as a skill, which we have the potential to achieve. We may question whether the various tests, in their current design, give boys and girls, and men and women, an equal basis for achieving their potential as readers. We now know that reading tests in schools are designed in a way that affects girls positively. We also have to question whether PIAAC reflects men’s reading skills more accurately than PIRLS and PISA, or whether the adult tests may be giving the men an advantage. This means that the challenge now is to find out how we can create reading tests that accurately demonstrate the actual skills of all boys and girls, and men and women, in terms of reading. That would give us a better basis for saying whether there really is reason to be concerned about boys’ reading skills,” concluded Solheim.