Thoughts on Exam Questions

Exam HallI was on the team that analysed UK GCSE Science questions from Summer 2008. It was a depressing experience - and our findings made for sorry stories in the media. In practice, the outcome should have surprised no-one; the Royal Society of Chemistry have been making warning noises for some time.

One of my colleagues on that team was Ken Zetie who is Head of Physics at St Paul's Boys School in London. He set one of the GCSE papers to his son, aged 8, with the inevitable discovery that, yes, even a child could do these exams.

He also reflected on the experience here.

One particular discovery we made was that a number of questions were accessible to candidates who had not even studied the subject. Other questions required little understanding - simply the ability to plug numbers into a formula.

Now consider that in some subjects, languages for example, candidates are allowed reference materials such as dictionaries.

Add a little bit of technology to that mix and I present you with a hierarchy of exam questions:

1. Can the question be done without the candidate having been to lessons at all ?

2. Can the question be done by simply rearranging information available from the exam paper itself (eg: plugging numbers into a given formula) ?

3. Can the question be done (in the time available) if the candidate has access to Wikipedia ?

4. Can the question be done (in the time available) if the candidate has access to WolframAlpha ? (If you haven't yet tried WolframAlpha then you really, really need to click on that link. Or this one).

5. Could you only do the question if you had a genuine understanding of the subject being examined ?

Wikipedia LogoWell, presumably ALL exam questions should be written to meet level 5 ?

Unfortunately, even a year on from our depressing experience, it seems the UK exam boards have yet to rise to the challenge. The latest "re-writes" are still regarded as too easy.

So there is still plenty to be done. And schools like my own will continue to use the International GCSEs in preference to the UK exams - an option that is finally being opened up to the UK state sector

But, looking to the future, what does this mean for the future of education in the face of Moore's Law and ever-improving "Knowledge Engines".

Time to think again.

Follow me on Twitter: @IanYorston

A peek at the future: Google Flu Trends

 This sort of thing is seriously smart. And a real indication of where Google can go by using the "Wisdom of the Crowds" as evidenced by activity passing through its own search engines.
Google Flu Trends.

Google have found that certain search terms are good indicators of flu activity. Google Flu Trends can use aggregated Google search data to estimate flu activity in US states up to two weeks faster than traditional systems.
This work is overseen by Google.org - the multi-billion dollar charitable arm that Google have established. They aim to use the power of information and technology to address the global challenges of our age: climate change, poverty and emerging disease. In collaboration with experienced partners working in each of these fields, Google plan to invest resources and tap the strengths of their employees and global operations to advance five major initiatives: .

Follow me on Twitter: @IanYorston

Not just football. Now we're dropping out of Science as well...

goalpost.jpgThis is an extraordinary story about the decline of British Education. But let me start on the subject of football. Because there are some interesting parallels.

Last week the various UK soccer sides all fell out of Euro 2008. Only 14 teams were able to qualify - and not one of our national sides made the cut. England promptly fired their national coach. Newspapers agonized over the decline and fall. There was much wailing and gnashing of teeth. This was, by all accounts, a national disaster.

This week the OECD published its Pisa survey. This is a comparative study of academic performance around the world. It publishes every three years and considers reading, maths and science skills. It too points to a national disaster.

The UK did VERY badly. Among 15-year-olds in 57 countries the UK ranked "between 12th and 18th place".

The organisers give a country's position as being ranked between certain positions because it says with a sample of students it is not always possible to state a comparative ranking with 100% accuracy. Instead, OECD calculates, with 95% confidence, a range of ranks that the country falls within.

So we can be 95% certain that UK school science is bad enough to see the national coach sacked.

Only that isn't quite how the BBC reported the story. Read on and be amazed:

UK among school science leaders

The UK is among the better performers in an international league table on school science

This is unbelievable reporting - not just rose-tinted spectacles but a full-blown case of myopia. Nor is this the first time that the BBC have reported an educational disaster as some curious form of triumph.

It obviously occurred to the BBC that people might - just might - check some of the facts. So the report continues with this extraordinary excuse:

In 2000, the UK was 4th, but the organisers say comparing results is not strictly valid because the tests have changed.

Note: not strictly valid. Hmm. I can't help thinking that a fall from 4th to 14th is sufficiently dramatic to be worthy of comparison despite some minor caveat.

The BBC then throws in this curious observation:

The UK as a whole was not included in the last Pisa study.

Well, yes, that's true - but a quick search of the BBC's own pages would reveal that in the last survey of 2003 the UK failed to provide enough data for the analysis of Maths and Science to be statistically valid (there's some deep irony in there somewhere).

So the last time the UK was properly assessed was back in 2000. And everything has been in decline ever since. The BBC reported that millennium survey in euphoric terms - and even managed to employ a footballing metaphor. Sadly the report is now most notable for the unfulfilled optimism of its closing paragraph.

Of course, it isn't only Science and Soccer that are in decline. Only last week it was announced that England had dropped from third place to 19th in the world in an assessment of reading.


FactFile

The Pisa survey is based on tests carried out in 2006 in 57 countries which together account for 90% of the world's economy. It tested students on how much they knew about science and their ability to use scientific knowledge to address questions in daily life.

Finland come out on top, followed by Hong Kong (China), Canada, Chinese Taipei, Estonia and Japan. Countries that have moved 'sharply upward' include Canada, Germany, Austria and Denmark.

Note that Estonia were part of the Soviet Union until 1991. Next thing you know, we'll be losing football games to Croatia...


Update

The BBC have now changed their article to better reflect the reality of the original Pisa Report. It now reads:

UK schools slip down in science

The UK is above average in a major international league table on school science - but it has slipped compared to its previous top-four ranking.

The whole article remains depressingly apologetic in tone - but it is at least a fairer reflection of the facts.
Follow me on Twitter: @IanYorston

Human metabolism recreated in lab

Software models become more and more sophisticated. The significance of this development is not simply that "real" tissue is no longer required but that the computational techniques now available allow for very fast analysis (faster than "real" time) and also "Darwinian" approaches which home in on "best" solutions by using evolutionary methods.

US researchers say they have created a "virtual" model of all the biochemical reactions that occur in human cells.

They hope the computer model will allow scientists to tinker with metabolic processes to find new treatments for conditions such as high cholesterol.

BBC NEWS | Health | Human metabolism recreated in lab.


Follow me on Twitter: @IanYorston

Sentient Developments: Must know terms for today's intelligentsia

I fell upon this quote from Carl Sagan whilst browsing Sentient Developments:

"The visions we offer our children shape the future. It matters what those visions are. Often they become self-fulfilling prophecies. Dreams are maps."

Which particularly struck me as I had only just finished listening to the latest podcast from the UK thinktank Demos.

Over at Demos (who are also in MySpace now), Hannah Green and Celia Hannon have been working on a project funded by the National College for School Leadership called Their Space: Education for a Digital Generation.

Paul Miller has been involved:

I helped out a bit with the research for the project and it’s been fascinating to work on. Hannah and Celia have done a brilliant job at bringing it all together and writing what I think is one of the best Demos reports for quite a while.

The piece basically takes apart the myth of ‘digital danger’ for teens. It suggests that schools in particular should have a lot more faith in kids ability to navigate the online world and should rearrange the way that IT is taught to put the kids in charge. The findings (which are all about kids in the UK) mesh neatly with things we’d learned about the US from danah boyd’s work.

The report caught my eye because the findings almost exactly mirror the talk I've been giving to Independent school audiences up and down the country for the last two years.

As ever, so much of the issue is encapsulated by Douglas Adams:

Anything that is in the world when you're born is normal and ordinary and is just a natural part of the way the world works.

Anything that's invented between when you're fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.

Anything invented after you're thirty-five is against the natural order of things.

Meanwhile - over at Sentient Developments

At the dawn of European humanism, Florentines believed that reading Dante while ignoring science was ridiculous. Similarly, Leonardo da Vinci and Michelangelo both recognized the great importance of understanding science, technology and engineering.

Despite these trail-blazers, not much has changed since then; a startling number of so-called 'intellectuals' remain grossly ignorant of pending technologies and the revealing sciences

They go on to offer a list of "must-know-terms" that includes the following:

  • accelerating change
  • augmented reality
  • human enhancement
  • molecular assembler
  • neural interface device
  • open source
  • participatory panopticon
  • political globalization
  • quantum computation
  • radical Luddism
  • remedial ecology
  • Simulation Argument
  • Singularity
  • ubiquitous surveillance
  • virtual reality

Great stuff. Go and read the full list. I'll test you later...

Follow me on Twitter: @IanYorston

Blinded by the Light

1997 saw the battle cry of Education, Education, Education. Well, here we are nearly 10 years later and things haven't got much further than the "vision thing" that Bill Gates once advocated.

In 1963, a previous Labour prime minister, Harold Wilson, called for a new Britain to be "forged in the white heat of this [technological] revolution".

Nearly half a century on, Tony Blair is to call for more of the same: more than ever, our economic future is through "the brilliant light of science".

The prime minister is a late convert to science.

[...]

Mr Blair has presided over a time where the numbers of young people studying physics and chemistry have dwindled by a fifth. And a quarter of schools have no qualified physics teachers.

This is a deficiency he acknowledges but says he's trying to put it right.

"We've got to invest in science far more as a country. The government is tripling investment in science - to recruit better science teachers - which is why we're offering all sorts of incentives for that to happen. We've got specialist science and technology colleges which we are creating."

However, investment is well short of the target set by the European Union's aim of being the "most competitive, dynamic knowledge-based economy in the world".

That's a statement from the EU's Lisbon Strategy which aims to match the US's research funding of about 3% of GDP by 2010.

Currently, Britain's is just over 1% with plans to increase to 2.5% by 2015.

Link: BBC Science/Nature.

Follow me on Twitter: @IanYorston

Unreasonable? Moi?

My wife has alerted me to the fact that if you type "unreasonable" into Google then this WebLog comes up as the top hit...

She is, she happily tells her friends, married to the most unreasonable person in the world - and she has Google to prove it.

She is gaining a lot of sympathy this way...

But every cloud has a silver lining.

Further down that list of Google results, I find an excellent article from 1993 by Prof. David Singmaster entitled The Unreasonable Utility of Recreational Mathematics.

In this he points out that:

  • Firstly, recreational problems are often the basis of serious mathematics.
  • Secondly, recreational mathematics has frequently turned up ideas of genuine but non-obvious utility.
  • Thirdly, recreational mathematics has great pedagogic utility.
  • Fourthly, recreational mathematics is very useful to the historian of mathematics.

And he concludes with the observation that "There really is considerable interest in mathematics out there and if we enjoy our subject, it should be our duty and our pleasure to try to encourage and feed this interest. Indeed, it may be necessary for our self-preservation."

Well worth a read.

Follow me on Twitter: @IanYorston