by Mike Aston, Consultant, UNESCO IITE

Mr. Aston's report provided a historical summary of the conduct of impact studies of ICT on education in the United Kingdom and Western European countries. He touched on the measures and indicators that evolved, current practices being and having been used, as well as specific experiences faced by European nations in the conduct of these studies. He also presented "a more sophisticated model for describing the nature of ICT impact at a national level."

The History of Measuring ICT Impact

The history of collecting meaningful data about the use of IT (ICT) in schools is relatively short. Microcomputers have been used in schools since 1979. The first survey carried out by the Department of Education and Science (DES) in the UK was in March 1990, the results being published as a Statistical Bulletin in June 1991. There is no mention of communications other than a note to say that modems were in use in several secondary schools for access to the national viewdata system (Prestel) and mainframe computers. The number of CD-ROM drives was noted as being on average of one for every ten secondary schools.

In 1989, Tim Berners-Lee, a scientist at the European Laboratory for Particle Physics in Geneva, had proposed the World Wide Web project. In 1992, The World Wide Web became accessible to the public and in the spring of 1993, a group of graduate students at the University of Illinois computer laboratories, led by 21-year-old Marc Andreessen, created a "browser" programme called Mosaic and distributed it free. Netscape and then Microsoft followed with browsers that greatly simplified a computer user's ability to search the Internet for information.

By 1999 there were 150 million users on the Internet, with over 800 million web pages accessible to anyone in the world with the necessary equipment and communications facility. By August 2001, the number of Internet users worldwide had risen to 513 million.

From 1991 to 1999, ICT in schools expanded rapidly in the developed nations through curriculum development, networking, the professional development of teachers and software improvements. Some nations also began ‘ICT in schools’ programmes, notably Chile, Costa Rica, India, Mauritius and Pakistan. During this period, the UK Ministry continued to measure ICT resources in school and other nations started to follow suite, with the Netherlands and Japan in the lead.

In 1996, a major provider of ICT resources to schools in the UK, Research Machines plc, commissioned The Advisory Unit: Computers in Education based in Hatfield, UK, to carry out a comparative study of ICT in schools in the G7 nations (Canada, France, Germany, Italy, Japan, the United Kingdom and the USA). The study concentrated on collecting quantitative data on methods used, including interviews (face-to-face and over the telephone), questionnaires, local and national government statistical reports, ministerial documents and personal contacts. For example, in Japan, the following institutions were contacted for data:

• National Ministry of Education (MONBUSHO)
• Government National Computing Commission
• The National Teacher Training University
• Hardware manufacturers
• Educational Software publishers
• Teacher Training Institutions
• Universities

Measuring the impact of ICT on school education systems was certainly not an exact science, as quantitative data often was ‘guesstimated’ or extrapolated from various sources with widely differing responses. Sometimes, official figures were clearly inflated in referring to figures obtained from end-users. Simple problems of defining a computer or the concept of ‘obsolescence’ in different mother tongues led to difficulties in obtaining accurate data. The biggest problem to overcome was that of timing: the study took more than three months to carry out, which was against the background of rapidly changing attitudes to ICT in education and technological advances - between them creating a moving set of targets. In many ways, the qualitative data collected on ICT in the curriculum and teachers’ take-up of the technology and software in teaching gave a more accurate picture of where a nation progressed. The RM G7 Report became known as the OrbIT Report and was repeated in 1998 and 2000, allowing year-on-year comparisons among nations.

From 1991 Tjeerd Plomp and Willem J Pelgrum at the Faculty of Educational Science & Technology, University of Twente in the Netherlands conducted studies on behalf of the International Association for the Evaluation of Educational Achievement (IEA) and the Organisation for Economic Co-operation and Development (OECD). The data was collected in various countries (the UK, Italy and most of Canada, for some reason did not contribute) by staff from National Project Centres, many of whom clearly did not have a background in ICT in education. Nevertheless, the studies tackled such issues as ‘Educating the Educators’ and ‘Structural Models of Implementation Indicators’.

Further on, the study lists key indicators and notes on their measurement:

• School size
• External financial support
• External training support
• Previous innovation experience
• Availability and resource needs
• Perceived innovation relevance
• School policy for computer use
• Internal staff development
• Internal innovation assistance
• Teacher competence and readiness
• Monitoring and problem coping strategy
• Implementation outcome

These earlier studies failed to predict the growth of local and wide area networking in schools, but were extremely rigorous in their statistical analysis.

The last study of note was carried out by the Directorate-General for Education and Culture in the European Commission 2000/2001 that resulted in two documents being published, namely
‘ICT@Europe.edu’ and ‘Basic Indicators on the Incorporation of ICT into European Education Systems – Facts and figures 2000/01 Annual Report’. The first booklet is entirely qualitative and covers the member states of the EU (15), EFTA/EEA (3) and the pre-accession countries (12). For each country, the report describes:

• Aims and strategies
• Sharing of responsibilities
• Public/private partnerships
• Major initiatives implemented

The second document is largely quantitative using bar charts or shaded maps of Europe to illustrate comparisons between nations on the National Policy and Basic indicators as listed below.

National Policy Descriptors:

1. Education levels covered by official documents on the use of ICT. Primary, secondary and higher education (in force during 2000/01)
2. National or official bodies with a remit for supervision and/or promotion of national policy for ICT in education, 2000/01
3. Implementation schedule for typical ICT projects. Primary and secondary education 1995-2010
4. Objectives in ICT. Primary education. Projects under way in 2000/01
5. Objectives in ICT. Secondary education. Projects under way in 2000/01

Basic Indicators:
6. Number of pupils per computer, and number of pupils per computer with an Internet connection. Primary education, 2001
7. Number of pupils per computer, and number of pupils per computer with an Internet connection. Secondary education, 2001
8. Responsibility for the purchase and maintenance of hardware. Primary and secondary education. Projects under way in 2000/01
9. Distribution of the specific budget between the purchase of equipment and expenditure on human resources. Primary education. Projects under way in 2000/01
10. Distribution of the specific budget between the purchase of equipment and expenditure on human resources. General lower secondary education. Projects under way in 2000/01
11. Distribution of the specific budget between the purchase of equipment and expenditure on human resources. General upper secondary education. Projects under way in 2000/01
12. Inclusion of ICT in the curriculum. Primary education 2000/01
13. Percentage of teachers who use computers and/or the Internet in the classroom. Primary education 2001
14. Average periods during which primary schoolteachers use computers (with or without Internet connections) in the classroom, in hours per week, 2001
15. Approaches to ICT defined in the curriculum. Primary education 2000/01
16. Objectives defined in the curriculum for the teaching or the use of ICT. Primary education, 2000/01
17. Inclusion of ICT in the curriculum. General secondary education 2000/01
18. Percentage of teachers who use computers and/or the Internet in the classroom. Secondary education 2001
19. Reasons given for not using the Internet with pupils. Secondary education 2001
20. Approaches to ICT defined in the curriculum. General lower secondary education 2000/01
21. Annual number of hours recommended for teaching ICT as a subject in its own right. General lower secondary education 2000/01
22. Objectives defined in the curriculum for the teaching or the use of ICT. General lower secondary 2000/01
23. Approaches to ICT defined in the curriculum. General upper secondary education 2000/01
24. Objectives defined in the curriculum for the teaching or the use of ICT. General upper secondary education 2000/01
25. Specialist ICT teachers. Primary and secondary education 2000/01
26. Inclusion of ICT in the initial training of all teachers (except specialist ICT teachers). Primary education, 2000/01
27. Inclusion of ICT in the initial training of all teachers (except specialist ICT teachers). Lower secondary education 2000/01
28. Inclusion of ICT in the initial training of all teachers (except specialist ICT teachers). General upper secondary education 2000/01
29. Percentage share of compulsory teaching related to ICT and the number of hours devoted to such teaching, in the initial training of all teachers (except specialist ICT teachers). Lower secondary level 2000/01
30. Desirable ICT skills according to official recommendations for the initial training of all teachers (except specialist ICT teachers). Lower secondary education 2000/01
31. Percentages of primary schoolteachers and secondary schoolteachers who have received official training in the use of computers and/or the Internet in their teaching, 2001
32. Percentages of primary schoolteachers and secondary schoolteachers in the EU who have received official training in the use of computers and/or the Internet in their teaching, 2001

This range of indicators offers a comprehensive picture of the incorporation of ICT into European Education Systems but is not judgmental in any way. There are a number of blanks (presumably not known) in the data sets and problems of aggregating data for countries like Germany (with 17 distinctive education systems), Spain (with 17 autonomous regions) and the UK (with 4 countries) results in extensive footnotes to the tables and maps.

Some Issues raised by the various studies published so far:

The three fundamental questions:

The Current Scenario

The UNESCO Institute of Information Technology in Education (IITE) situated in Moscow has initiated a major project to determine ways of measuring ICT impact on national education systems using a series of indicators. An Expert Meeting was held in Moscow during March 2001 with representatives from such widely dispersed countries as Namibia, Pakistan and the state of Brandenburg (Germany). It was agreed to test a set of indicators with a number of countries in the Commonwealth of Independent States (CIS), formerly part of the USSR, but the results from this pilot trial have yet to be published.

The IITE also initiated a pilot seminar held in Moscow for High Level Decision-Makers in ICT Educational Policy during 2001. 19 states (mainly from the CIS) were represented and as part of the development of materials for the seminar two indicator matrices were created. The first, subsequently become known as Morel’s Matrix, after Professor Raymond Morel from Switzerland, one of the tutors on the seminar. This, in essence, presents four levels of development, namely Emerging, Applying, Integrating and Transforming. The theory is that a nation’s ICT development in its education system can be mapped onto the matrix and can be described in general terms such as ‘the majority of the criteria have reached the Applying phase but with X still Emerging and Y already Integrating.

Applied "Morel's Matrix"

Criteria/Phase	Emerging	Applying	Integrating	Transforming
Context
Pedagogy
Vision
Coverage of ICT Strategy
Curriculum
Resources
Staff Development
Quality Assurance

 

The second matrix, attributed to Matti Sinko (Finland) and Mike Aston (UK), also tutors on the seminar in Moscow, focuses on National Priorities in the Short (2 years) and Long (3-5 years) terms. The two time periods are further divided into Producing and Updating/Revising phases. The matrix is completed by the National or Regional ICT in Education Strategy leaders or Policy makers and the following areas are covered in the matrix:

General Rationale & Aims Objectives Methods of implementation Funding plan Quality Assurance and Evaluation, Indicators Focus on primary phase Focus on secondary phase Focus on tertiary phase Focus on adult education Special needs Capacity building - Pre-service teacher education - Staff development Management - national level - local level - institutional level Quality Assurance, Inspection Research

Curriculum & pedagogical development ICT as a subject ICT across the curriculum Vocational ICT training Software Generic software tools Subject specific software and digital materials Localising, versioning, licensing Production, delivery, updating and use Support - Technical - Pedagogical - Media design Technical infrastructure Hardware Local-area networks Wide-area networks (Internet) Student assessment Other, please, specify

 

Proposed Model for the Future

There is a clear need to collect both quantitative and qualitative data on ICT development in different nations and be able to present it in a reasonable concise manner. A combination of the EU data presentation, the IITE questionnaire, Morel’s matrix and the Aston/Sinko matrix would provide a comprehensive set of indicators. The next stage is to synthesise the data collection methods and start the process.

Almost every country/region is embarked on a process of significant change in school education as a result of the all-pervasive nature of ICT. The advent of the Internet offers a mechanism for nations and regions to avoid re-invention, to learn from each other and to relate to different circumstances and economic resourcing. UNESCO is a strong contender to act as a clearing-house for this level of dynamic information.

References

Pelgrum, Willem J., and Tjeerd Plomp The Use of Computers in Education Worldwide: Results from the IEA Computers in Education Survey in 19 Education Systems. Oxford: Pergamon Press. 1991

Pelgrum, W.J. & Plomp, Tj. (Eds). The IEA Study of Computers in Education: Implementation of an Innovation in 21 Educational Systems. Oxford: Pergamon Press. 1993

Schools, Teachers and Computers: a Cross-National Perspective. Preliminary report, IEA-Comped Study Stage 2. International Association for the Evaluation of Educational Achievement, University of Twente, Centre for Applied Educational Research, Enschede, The Netherlands. 1993

Plomp, Tj., Anderson, R.E. & Kontogiannopoulou, G. (Eds). Cross National Policies and Practices on Computers in Education. Dordrecht (the Netherlands): Kluwer Academic Publishers Group. 1996

The RM G7 Report Eds. Aston M. and Hinder A. The Advisory Unit: Computers in Education, Hatfield UK 1996

The RM OrbIT G8 + 6 other nations Report Eds. Aston M. and Hinder A. The Advisory Unit: Computers in Education, Hatfield UK 1998

The RM OrbIT G8 Report + 14 other nations Report Eds. Aston M. and Hinder A. The Advisory Unit: Computers in Education, Hatfield UK 2000

Directorate-General for Education and Culture European Commission ICT@Europe.edu publ. Eurydice European Unit, Brussels ISBN 2-87116-324-3 2001

Directorate-General for Education and Culture European Commission Basic Indicators on the Incorporation of ICT into European Education Systems Facts and Figures publ Eurydice European Unit, Brussels ISBN 2-87116-332-4 2001

ICTs in Education: State-of-the-Art, Needs and Perspectives – Indicators and Information Systems UNESCO IITE Moscow 2001