definition of evaluation by different authors

(2007), Nason et al. Impact is assessed alongside research outputs and environment to provide an evaluation of research taking place within an institution. Table 1 summarizes some of the advantages and disadvantages of the case study approach. Definition of evaluation. trailer << /Size 97 /Info 56 0 R /Root 61 0 R /Prev 396309 /ID[<8e25eff8b2a14de14f726c982689692f><7a12c7ae849dc37acf9c7481d18bb8c5>] >> startxref 0 %%EOF 61 0 obj << /Type /Catalog /Pages 55 0 R /Metadata 57 0 R /AcroForm 62 0 R >> endobj 62 0 obj << /Fields [ ] /DR << /Font << /ZaDb 38 0 R /Helv 39 0 R >> /Encoding << /PDFDocEncoding 40 0 R >> >> /DA (/Helv 0 Tf 0 g ) >> endobj 95 0 obj << /S 414 /T 529 /V 585 /Filter /FlateDecode /Length 96 0 R >> stream The exploitation of research to provide impact occurs through a complex variety of processes, individuals, and organizations, and therefore, attributing the contribution made by a specific individual, piece of research, funding, strategy, or organization to an impact is not straight forward. Key features of the adapted criteria . Attempts have been made to categorize impact evidence and data, for example, the aim of the MICE Project was to develop a set of impact indicators to enable impact to be fed into a based system. 0000012122 00000 n An alternative approach was suggested for the RQF in Australia, where it was proposed that types of impact be compared rather than impact from specific disciplines. An evaluation essay is a composition that offers value judgments about a particular subject according to a set of criteria. From the outset, we note that the understanding of the term impact differs between users and audiences. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. In many instances, controls are not feasible as we cannot look at what impact would have occurred if a piece of research had not taken place; however, indications of the picture before and after impact are valuable and worth collecting for impact that can be predicted. In development of the RQF, The Allen Consulting Group (2005) highlighted that defining a time lag between research and impact was difficult. n.d.). Different authors have different notions of educational evaluation. This might describe support for and development of research with end users, public engagement and evidence of knowledge exchange, or a demonstration of change in public opinion as a result of research. This work was supported by Jisc [DIINN10]. In the UK, UK Department for Business, Innovation, and Skills provided funding of 150 million for knowledge exchange in 201112 to help universities and colleges support the economic recovery and growth, and contribute to wider society (Department for Business, Innovation and Skills 2012). A very different approach known as Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions (SIAMPI) was developed from the Dutch project Evaluating Research in Context and has a central theme of capturing productive interactions between researchers and stakeholders by analysing the networks that evolve during research programmes (Spaapen and Drooge, 2011; Spaapen et al. This is recognized as being particularly problematic within the social sciences where informing policy is a likely impact of research. << /Length 5 0 R /Filter /FlateDecode >> The Goldsmith report concluded that general categories of evidence would be more useful such that indicators could encompass dissemination and circulation, re-use and influence, collaboration and boundary work, and innovation and invention. The inherent technical disparities between the two different software packages and the adjustment . In the UK, evaluation of academic and broader socio-economic impact takes place separately. SIAMPI has been used within the Netherlands Institute for health Services Research (SIAMPI n.d.). It has been acknowledged that outstanding leaps forward in knowledge and understanding come from immersing in a background of intellectual thinking that one is able to see further by standing on the shoulders of giants. To adequately capture interactions taking place between researchers, institutions, and stakeholders, the introduction of tools to enable this would be very valuable. A variety of types of indicators can be captured within systems; however, it is important that these are universally understood. One might consider that by funding excellent research, impacts (including those that are unforeseen) will follow, and traditionally, assessment of university research focused on academic quality and productivity. 0000334705 00000 n %PDF-1.4 % Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment. While aspects of impact can be adequately interpreted using metrics, narratives, and other evidence, the mixed-method case study approach is an excellent means of pulling all available information, data, and evidence together, allowing a comprehensive summary of the impact within context. Media coverage is a useful means of disseminating our research and ideas and may be considered alongside other evidence as contributing to or an indicator of impact. (2011) Maximising the Impacts of Your Research: A Handbook for Social Scientists (Pubd online) <, Lets Make Science Metrics More Scientific, Measuring Impact Under CERIF (MICE) Project Blog, Information systems of research funding agencies in the era of the Big Data. What are the reasons behind trying to understand and evaluate research impact? 0000011585 00000 n 0000009507 00000 n For example, the development of a spin out can take place in a very short period, whereas it took around 30 years from the discovery of DNA before technology was developed to enable DNA fingerprinting. As a result, numerous and widely varying models and frameworks for assessing impact exist. Many theorists, authors, research scholars, and practitioners have defined performance appraisal in a wide variety of ways. It is worth considering the degree to which indicators are defined and provide broader definitions with greater flexibility. A university which fails in this respect has no reason for existence. Research findings including outputs (e.g., presentations and publications), Communications and interactions with stakeholders and the wider public (emails, visits, workshops, media publicity, etc), Feedback from stakeholders and communication summaries (e.g., testimonials and altmetrics), Research developments (based on stakeholder input and discussions), Outcomes (e.g., commercial and cultural, citations), Impacts (changes, e.g., behavioural and economic). 5. The REF will therefore assess three aspects of research: Research impact is assessed in two formats, first, through an impact template that describes the approach to enabling impact within a unit of assessment, and second, using impact case studies that describe the impact taking place following excellent research within a unit of assessment (REF2014 2011a). The Payback Framework enables health and medical research and impact to be linked and the process by which impact occurs to be traced. Hb```f``e`c`Tgf@ aV(G Ldw0p)}c4Amff0`U.q$*6mS,T",?*+DutQZ&vO T4]2rBWrL.7bs/lcx&-SbiDEQ&. , , . The current definition of health, formulated by the WHO, is no longer adequate for dealing with the new challenges in health care systems. Measurement assessment and evaluation helps the teachers to determine the learning progress of the students. From an international perspective, this represents a step change in the comprehensive nature to which impact will be assessed within universities and research institutes, incorporating impact from across all research disciplines. 2007). 2009; Russell Group 2009). If metrics are available as impact evidence, they should, where possible, also capture any baseline or control data. To enable research organizations including HEIs to monitor and manage their performance and understand and disseminate the contribution that they are making to local, national, and international communities. It is now possible to use data-mining tools to extract specific data from narratives or unstructured data (Mugabushaka and Papazoglou 2012). In the UK, evidence and research impacts will be assessed for the REF within research disciplines. Citations (outside of academia) and documentation can be used as evidence to demonstrate the use research findings in developing new ideas and products for example. Aspects of impact, such as value of Intellectual Property, are currently recorded by universities in the UK through their Higher Education Business and Community Interaction Survey return to Higher Education Statistics Agency; however, as with other public and charitable sector organizations, showcasing impact is an important part of attracting and retaining donors and support (Kelly and McNicoll 2011). Perhaps the most extended definition of evaluation has been supplied by C.E.Beeby (1977). 0000002109 00000 n Figure 2 demonstrates the information that systems will need to capture and link. In designing systems and tools for collating data related to impact, it is important to consider who will populate the database and ensure that the time and capability required for capture of information is considered. n.d.). The Payback Framework systematically links research with the associated benefits (Scoble et al. Research findings will be taken up in other branches of research and developed further before socio-economic impact occurs, by which point, attribution becomes a huge challenge. In education, the term assessment refers to the wide variety of methods or tools that educators use to evaluate, measure, and document the academic readiness, learning progress, skill acquisition, or educational needs of students. More details on SROI can be found in A guide to Social Return on Investment produced by The SROI Network (2012). Test, measurement, and evaluation are concepts used in education to explain how the progress of learning and the final learning outcomes of students are assessed. 2007). Without measuring and evaluating their performance, teachers will not be able to determine how much the students have learned. 6. One of these, the RQF, they identified as providing a promising basis for developing an impact approach for the REF using the case study approach. Capturing data, interactions, and indicators as they emerge increases the chance of capturing all relevant information and tools to enable researchers to capture much of this would be valuable. 0000007967 00000 n 2007). The Oxford English Dictionary defines impact as a 'Marked effect or influence', this is clearly a very broad definition. Assessment for learning is ongoing, and requires deep involvement on the part of the learner in clarifying outcomes, monitoring on-going learning, collecting evidence and presenting evidence of learning to others.. Decker et al. By allowing impact to be placed in context, we answer the so what? question that can result from quantitative data analyses, but is there a risk that the full picture may not be presented to demonstrate impact in a positive light? Impact assessments raise concerns over the steer of research towards disciplines and topics in which impact is more easily evidenced and that provide economic impacts that could subsequently lead to a devaluation of blue skies research. Every piece of research results in a unique tapestry of impact and despite the MICE taxonomy having more than 100 indicators, it was found that these did not suffice. Understand. 0000334683 00000 n Definition of Evaluation "Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education, as part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have." Mary Thorpe 2. One notable definition is provided by Scriven (1991) and later adopted by the American Evaluation Association (): "Evaluation is the systematic process to determine merit, worth, value, or . The time lag between research and impact varies enormously. In viewing impact evaluations it is important to consider not only who has evaluated the work but the purpose of the evaluation to determine the limits and relevance of an assessment exercise. What are the challenges associated with understanding and evaluating research impact? The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. And also that people who are recognized as authors, understand their responsibility and accountability for what is being published. The range and diversity of frameworks developed reflect the variation in purpose of evaluation including the stakeholders for whom the assessment takes place, along with the type of impact and evidence anticipated. Even where we can evidence changes and benefits linked to our research, understanding the causal relationship may be difficult. Author: HPER Created Date: 3/2/2007 10:12:16 AM . Findings from a Research Impact Pilot, Institutional Strategies for Capturing Socio-Economic Impact of Research, Journal of Higher Education Policy and Management, Introducing Productive Interactions in Social Impact Assessment, Measuring the Impact of Publicly Funded Research, Department of Education, Science and Training, Statement on the Research Excellence Framework Proposals, Handbook on the Theory and Practice of Program Evaluation, Policy and Practice Impacts of Research Funded by the Economic Social Research Council. If knowledge exchange events could be captured, for example, electronically as they occur or automatically if flagged from an electronic calendar or a diary, then far more of these events could be recorded with relative ease. 60 0 obj << /Linearized 1 /O 63 /H [ 1325 558 ] /L 397637 /E 348326 /N 12 /T 396319 >> endobj xref 60 37 0000000016 00000 n At least, this is the function which it should perform for society. Johnston (Johnston 1995) notes that by developing relationships between researchers and industry, new research strategies can be developed. The definition problem in evaluation has been around for decades (as early as Carter, 1971), and multiple definitions of evaluation have been offered throughout the years (see Table 1 for some examples). HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). n.d.). Evaluation research aimed at determining the overall merit, worth, or value of a program or policy derives its utility from being explicitly judgment-oriented. 2007) who concluded that the researchers and case studies could provide enough qualitative and quantitative evidence for reviewers to assess the impact arising from their research (Duryea et al. The growing trend for accountability within the university system is not limited to research and is mirrored in assessments of teaching quality, which now feed into evaluation of universities to ensure fee-paying students satisfaction. different things to different people, and it is primarily a function of the application, as will be seen in the following. It can be seen from the panel guidance produced by HEFCE to illustrate impacts and evidence that it is expected that impact and evidence will vary according to discipline (REF2014 2012). 2006; Nason et al. They risk being monetized or converted into a lowest common denominator in an attempt to compare the cost of a new theatre against that of a hospital. 0000006922 00000 n 0000007777 00000 n Downloadable! This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. Figure 1, replicated from Hughes and Martin (2012), illustrates how the ease with which impact can be attributed decreases with time, whereas the impact, or effect of complementary assets, increases, highlighting the problem that it may take a considerable amount of time for the full impact of a piece of research to develop but because of this time and the increase in complexity of the networks involved in translating the research and interim impacts, it is more difficult to attribute and link back to a contributing piece of research. Evaluation is a process which is continuous as well as comprehensive and involves all the tasks of education and not merely tests, measurements, and examination. In terms of research impact, organizations and stakeholders may be interested in specific aspects of impact, dependent on their focus. A taxonomy of impact categories was then produced onto which impact could be mapped. For full access to this pdf, sign in to an existing account, or purchase an annual subscription. However, the . Systems need to be able to capture links between and evidence of the full pathway from research to impact, including knowledge exchange, outputs, outcomes, and interim impacts, to allow the route to impact to be traced. The University and College Union (University and College Union 2011) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. According to Hanna- " The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". To achieve compatible systems, a shared language is required. This is being done for collation of academic impact and outputs, for example, Research Portfolio Online Reporting Tools, which uses PubMed and text mining to cluster research projects, and STAR Metrics in the US, which uses administrative records and research outputs and is also being implemented by the ERC using data in the public domain (Mugabushaka and Papazoglou 2012). Studies (Buxton, Hanney and Jones 2004) into the economic gains from biomedical and health sciences determined that different methodologies provide different ways of considering economic benefits. Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. The Social Return on Investment (SROI) guide (The SROI Network 2012) suggests that The language varies impact, returns, benefits, value but the questions around what sort of difference and how much of a difference we are making are the same. 0000003495 00000 n This report, prepared by one of the evaluation team members (Richard Flaman), presents a non-exhaustive review definitions of primarily decentralization, and to a lesser extent decentralization as linked to local governance. Teacher Education: Pre-Service and In-Service, Introduction to Educational Research Methodology, Teacher Education: Pre-Service & In-Service, Difference and Relationship Between Measurement, Assessment and Evaluation in Education, Concept and Importance of Measurement Assessment and Evaluation in Education, Purpose, Aims and Objective of Assessment and Evaluation in Education, Main Types of Assessment in Education and their Purposes, Main Types of Evaluation in Education with Examples, Critical Review of Current Evaluation Practices B.Ed Notes, Compare and Contrast Formative and Summative Evaluation in Curriculum Development B.ED Notes, Difference Between Prognostic and Diagnostic Evaluation in Education with Examples, Similarities and Difference Between Norm-Referenced Test and Criterion-Referenced Test with Examples, Difference Between Quantitative and Qualitative Evaluation in Education, Difference between Blooms Taxonomy and Revised Blooms Taxonomy by Anderson 2001, Cognitive Affective and Psychomotor Domains of Learning Revised Blooms Taxonomy 2001, Revised Blooms Taxonomy of Educational Objectives, 7 Types and Forms of Questions with its Advantages, VSA, SA, ET, Objective Type and Situation Based Questions, Definition and Characteristics of Achievement Test B.Ed Notes, Steps, Procedure and Uses of Achievement Test B.Ed Notes, Meaning, Types and Characteristics of diagnostic test in Education B.ED Notes, Advantages and Disadvantages of Diagnostic Test in Education B.ED Notes, Types of Tasks: Projects, Assignments, Performances B.ED Notes, Need and Importance of CCE: Continuous and Comprehensive Evaluation B.Ed Notes, Characteristics & Problems Faced by Teachers in Continuous and Comprehensive Evaluation, Meaning and Construction of Process Oriented Tools B.ED Notes, Components, Advantages and Disadvantages of Observation Schedule, Observation Techniques of Checklist and Rating Scale, Advantages and Disadvantages of Checklist and Rating Scale, Anecdotal Records Advantages and Disadvantages B.ED Notes, Types and Importance of Group Processes and Group Dynamics, Types, Uses, Advantages & Disadvantages of Sociometric Techniques, Stages of Group Processes & Development: Forming, Storming, Norming, Performing, Adjourning, Assessment Criteria of Social Skills in Collaborative or Cooperative Learning Situations, Portfolio Assessment: Meaning, Scope and Uses for Students Performance, Different Methods and Steps Involved in Developing Assessment Portfolio, Characteristics & Development of Rubrics as Tools of Assessment, Types of Rubrics as an Assessment Tool B.ED Notes, Advantages and Disadvantages of Rubrics in Assessment, Types & Importance of Descriptive Statistics B.ED Notes, What is the Difference Between Descriptive and Inferential Statistics with Examples, Central Tendency and Variability Measures & Difference, What are the Different Types of Graphical Representation & its importance for Performance Assessment, Properties and Uses of Normal Probability Curve (NPC) in Interpretation of Test Scores, Meaning & Types of Grading System in Education, Grading System in Education Advantages and Disadvantages B.ED Notes, 7 Types of Feedback in Education & Advantages and Disadvantages, Role of Feedback in Teaching Learning Process, How to Identify Learners Strengths and Weaknesses, Difference between Assessment of Learning and Assessment for Learning in Tabular Form, Critical Review of Current Evaluation Practices and their Assumptions about Learning and Development, The Concept of Test, Measurement, Assessment and Evaluation in Education. Indicators were identified from documents produced for the REF, by Research Councils UK, in unpublished draft case studies undertaken at Kings College London or outlined in relevant publications (MICE Project n.d.). SROI aims to provide a valuation of the broader social, environmental, and economic impacts, providing a metric that can be used for demonstration of worth. One way in which change of opinion and user perceptions can be evidenced is by gathering of stakeholder and user testimonies or undertaking surveys. 2009). It is concerned with both the evaluation of achievement and its enhancement. It is acknowledged in the article by Mugabushaka and Papazoglou (2012) that it will take years to fully incorporate the impacts of ERC funding. A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006). Accountability. The ability to record and log these type of data is important for enabling the path from research to impact to be established and the development of systems that can capture this would be very valuable.