Audit of the NAESU: conditions for public funding

Procedures and Criteria of Evaluation of the Quality of Research Activities at the NAESU

According to the NAESU Regulations of the planning and implementation of research, in order to start a new research project at the NAESU, you need to participate in a competition which evaluates the relevance and usefulness of the suggested topic, the possibility of practical implementation of study results, and the authors’ previous research experience. Each study that wins the competition must be accountable and regularly monitored, and fundamental research projects are also required to undergo state expert evaluation to determine their scientific level. The Regulations mention that any study can be terminated if it is not implemented on the proper level, although the criteria of the proper level are not articulated. The decision to terminate a research project is made by the head of the institution with the department’s approval. The authors must report on the results of every stage of their research project, and after they finish the study, they must produce the final annotated report. The intermediate research findings are also recorded in the annual reports of the institution.

So the research projects at the NAESU are, according to the Regulations, regularly evaluated through reporting. At the same time, each project must meet the requirements indicated in its agreement or terms of reference. The finished studies are also evaluated based on the reporting documents submitted by their authors. These documents may include, in addition to the scientific report itself, any kind of study materials, electronic resources, etc. The authors may also attach any MES certificates (if they have created any pedagogical products), agreements about the implementation of research findings, patents or copyright certificates to their reports.

Each research project must undergo an expert review, but the person responsible for the review is the supervisor of the research project, which may cause a conflict of interest. In the interviews, NAESU representatives noted that there is both internal and external review process, and both elements are mandatory. However, the Regulations do not state this requirement. Reviews, reports and all the other materials are submitted by the supervisor to the Scientific Council of the institution, which, based on the materials and the oral report, decides whether the research project has been implemented and whether its results should be recommended for publishing. If the Council decides that the research project has not been implemented, the topic supervisor or the implementer of the research can request independent expert evaluation. If the expert evaluation also confirms the negative assessment of the study, the employment contract with its author can be terminated early.

The manuscripts written about the studies undergo additional expert evaluation which serves as a basis for the decision to include them in the annual printing plan. The evaluation is carried out by the department’s expert council, a permanent advisory body whose members are department employees; however, external researchers can also be involved in the expert evaluation. According to the Regulations of the Procedure of Planning and Control Over the Implementation of Research Projects, certain topics can also be considered by the Presidium. “The most important scientific results of research by the subordinate institutions are heard by the Presidium of the NAES of Ukraine.”

In the public reports of the NAESU published before 2017 there is little information about the quality of research. Many study results are labelled “significant” without explaining where this conclusion comes from. The chapter on the “Implementation of Research Findings” contains mostly statistical data about the number of analytical materials and publications based on the studies, as well as the number of educational institutions and government bodies which “implemented” these publications (it is not clear from the reports if schools actually do implement the methodological developments, or if they just receive printed copies of the publications). The NAESU also publishes collections of “Scientific Developments by the NAESU, recommended for implementation,” which contain lists of publications — both scientific and methodological — published at the NAESU. Nearly every publication is accompanied by an explanation how its implementation will promote something. The attention to the implementation of research findings demonstrates that the NAESU pays attention, at least formally, to the practical significance of its developments.

One of the respondents from an IPPE noted that there is no evaluation of the effectiveness of the implementation of NAESU research, and this complicates public policy development, particularly the development of state standards:

“I mean, obviously, there was some kind of research at some point, it did happen on a certain level. I mean, there is some kind of methodological concept, some educational strategies or something, which served as a basis for a certain approach, a certain line of textbooks and so on. But then this level of implementation became self-sufficient. And there was no deeper research, we rarely see any evaluation of the effectiveness of a certain approach or methodological toolkit which is based on that approach.”

That is, even if a study was implemented by printing methodological recommendations or textbooks, the quality of these products and their use is not evaluated.

Under these circumstances, the implementation itself, without any evaluation of its results or impact, cannot definitively say anything about the quality of the NAESU scientific developments.

Since 2017, the NAESU reports have included indicators of research quality (in particular, they were listed in the report on 2016). These indicators include “the average citation index of scientists,” “the average impact factor of scientific publications” and “the number of NAESU journals which are included in citation databases,” “the number of publications in foreign journals,” etc. However, neither the reports nor the NAESU’s Strategy of Development for 2016-22, which introduced these indicators, explain their meaning in more detail — for example, they do not name the citation index they use and the citation databases they mean (not all databases have strict criteria to ensure that they index only peer-reviewed scientific journals). Nevertheless, these indicators mark significant progress compared to the older reports.

Evaluation of Publications of the NAESU Employees in Terms of Meeting the Basic Requirements for Scientific Articles

The evaluation of whether the papers published by three NAESU institutes meet the basic requirements for scientific papers (see Methodology) produced slightly different results when carried out by the two experts, because they used different approaches. In the case of the criterion “Does the literature review include an analysis of studies on the paper’s topic carried out by other authors in the past?”, the CEDOS expert took into account that an overview article can contain analysis of previous work in the main body of the article, so it may satisfy this criterion, even if there is no analysis in the chapter specifically dedicated to the analysis of latest research. In addition, the CEDOS expert decided that historical papers which use primary sources (such as archival materials) satisfy the criterion “Does the publication use any data processed by the author?” The independent expert had a narrower interpretation of these two criteria. However, he interpreted the criterion “Does the paper explain the novelty of the study?” more widely, so in his analysis, more NAESU articles meet this criterion. The independent expert was also more strict about the question “Does the journal in which the paper is published have a clearly defined subject field?”, because he specializes in scientific publishing.

The results of the analyses by both experts for all the criteria are listed in the Table below. When the criterion is conditional (starts with the word “If”), then the table shows two values: the fraction among the papers that satisfy the condition and the fraction among all the analyzed papers.

Because of the different approaches and the impossibility of studying all questions in detail (for example, it is not always easy to find websites of Ukrainian journals), the Table sometimes demonstrates significant discrepancies. However, in the criteria of novelty, methodology and data usage the differences between experts are insignificant. As the final grade for all the criteria, we picked the one which is more favorable for the NAESU (emphasized with bold font in the table).

Table I.2.1. Comparison of the results of evaluation of whether papers by NAESU researchers meet the basic requirements for scientific papers.

Therefore, based on the evaluation of the NAESU papers in terms of satisfying the scientific standards, we can make the following generalizations:

• Only 28.5% of the articles contain analysis of previous research on the topic, in any form. Although most papers have a chapter dedicated to the analysis of recent research, this chapter usually contains only a list of names of the scientists who have worked on the topic, without references to the relevant work by these scientists. Such “analysis” does not let the reader know if the article’s author even read other papers on his or her topic. Among the names listed in such analyses, there are usually many other NAESU researchers, while mentions of contemporary foreign authors are extremely rare.

• 78% of the literature analyses and 75% of the lists of names (which are used as substitutes for literature analyses) mention contemporary authors. This percentage is logical, given the high number of the authors’ contemporary NAESU colleagues in the lists of names.

• 62% of the literature analyses and lists of names mention only Soviet and post-Soviet authors. In many other cases Western researchers are represented by only two or three names of the old masters of psychology and pedagogy (Montessori, Piaget, etc.). The analyzed articles allow us to understand that even though 27 years have passed since Ukraine gained independence, Ukrainian educational science still rarely remembers that foreign educational research exists. The exceptions are mostly papers written at the Institute of Higher Education and at the Department of Comparative Pedagogy of the Institute of Pedagogy.

• Only 23.6% of all papers contain mentions of literature which is neither in Ukrainian nor in Russian. Even the papers written in English often mention exclusively post-Soviet literature written in Ukrainian or Russian.

• Only 10.8% of the papers explain what the novelty of the material is. This fact in and of itself does not mean that 90% of the articles do not carry any novelty. However, it means that the NAESU researchers usually are not instructed to produce qualitatively new scientific knowledge and not encouraged to do it by the journals’ editorial policies. The chapters on “formulating the purpose of the article” in many papers contain phrases such as “to outline the problem” or “to clarify the concept,” without any intention to contribute anything to scientific knowledge about education. Thus, the majority of papers resemble student overviews rather than scientific articles. The small word count of the overwhelming majority of the papers (they are rarely longer than 10 pages) and the short reference list (rarely more than 15 points) only increases the resemblance to student papers.

• Only 8% of the articles contain descriptions of the research method. If the methodology is explained, it allows the reader to understand how the scientist reached his or her conclusions described in the paper. The NAESU articles usually do not provide this explanation, although it is to be expected for such practical and empirical discipline as pedagogy.

• 17% of the papers contain data which were processed and analyzed by the author to some extent. Almost a half of these papers are articles about the history of pedagogy, which use many old publications and archival materials. The NAESU papers rarely analyse any experiments or surveys conducted by authors, or any statistical data, although many Western journals (particularly in the field of education) do not even consider papers which do not mention any results of empirical research. Unfortunately, even those NAESU articles which described some results of original empirical studies often dedicated only two or three pages to those results, without any detailed analysis.

• As for the journals, in 15.7% of all cases the paper’s author is a member of the editorial board, which indicates a possible conflict of interest. The NAESU materials are mostly published in thematic pedagogical journals, but 17% of the articles are published in journals which have extremely wide topic range. This is especially the case for the papers in English, which were published, for example, in the Canadian Journal of Science, Education and Culture, the Modern Science journal, the Science Rise journal, and even the Oxford Journal of Scientific Research (no connection to the Oxford university). None of these journals are indexed in the Scopus database.

• Only in 29% of all cases the experts managed to find any information about the review procedures at the journals (it is, of course, impossible to check whether these procedures are actually followed). However, the articles published in the non-peer-reviewed journals generally do not look fundamentally different than the articles in the peer-reviewed journals.

The analysis covers three years, in which changes could have happened. However, the dynamics of these indicators over time does not allow us to see any significant tendencies of the scientific quality standards to improve or to deteriorate. The percentage of articles which mentioned only post-Soviet authors in their literature reviews fell from 66% in 2014 to 60% in 2015 and 2016; the methodology of research was described in 6.7% of all papers in 2014 and in 11% of all papers in 2016. These are the only even slightly noticeable shifts which we managed to record based on our analysis.

Therefore, the majority of the analyzed articles in some aspects fail to meet the high standards of scientific papers which could be published in peer-reviewed journals in Europe or North America. We must particularly emphasize such negative phenomena as the authors’ tendency to list the names instead of analyzing the past research; the neglect of almost all Western research; as well as the virtual absence of empirical work with data.

Use of the Results of the NAESU Research Activities in the Work of the Ministry of Education and Science and its Institutions

The MES and its subordinate institutions involved in the development and implementation of government policies in education are some of the key partners of the NAESU and beneficiaries of its activities. Based on the results of in-depth interviews and the analysis of the cooperation plans, we can conclude that the NAESU most actively interacts with the MES itself and with the IMCE.

In-depth interviews with MES employees demonstrated that the experience of cooperation with the NAESU and of using the results of its research differed depending on the position and the sphere in which the respondents worked (most of all on the educational level in which they developed policies). NAESU workers are invited to participate in all the official working groups of the MES and its institutions, because the MES Regulations require it, but their actual contribution depends on the expert level of the particular person. The NAESU also carries out expert evaluation for the MES, particularly the evaluation of state educational standards.

There is a stable connection between the MES and NAESU institutions and their departments. The connection manifests either in informal personal contacts of MES employees who are familiar with the topics on which the NAESU institutions or individual NAESU researchers work and who talk directly to the specialists they need, or in official requests from the MES in the cases when fulfilling the request takes a lot of time, and the researchers need to insert it in their working plans. However, judging from the interviews, the first type of cooperation is more prevalent.

Some respondents from the MES (working with secondary, higher and inclusive education) said that they used NAESU research findings in their daily work. They emphasized that they used practical research which can be used in the development of methodological recommendations, study plans, analytical or informational notes; they also said they read NAESU periodicals — again, of methodological rather than scientific kind. The respondents working with other levels of education do not use NAESU research in their work.

Some respondents from the MES noted that they are familiar with the work of some NAESU researchers, but that the general level of research and the professional competence of the employees at NAESU institutions is not very high, and the results of the NAESU activities do not meet the respondents’ expectations of quality. In addition, the respondents noted that many results of the NAESU research activities are “theoretical” and therefore inapplicable in their work, both because of their length and complicated writing style and because their format does not correspond to the needs of policy making in education. The MES representatives noted that the NAESU does not inform them about its work and does not sufficiently communicate the results of its research in a form which would be convenient to understand and use. However, another possible reason for the fact that MES representatives are not very familiar with the NAESU research is that the general practice of using research to make policy decisions in education has not been adopted in Ukraine.

In general, current decision making, according to one of the respondents, happens within the MES — NAESU — IMCE triangle. At each of these institutions, the employees who have similar areas of responsibility or research interests actually cooperate and interact. As for the higher-level policies (and most of all the drafting of bills), the NAESU participates in their making through its representatives in working groups, but these representatives are a rather small group of scientists with the highest status (academicians, Presidium members). However, as a MES representative noted,

“We make decisions based on experience, intuition, some kind of general understanding, general tendencies, rather than specific research…”

At the same time, the need for research which could serve as a basis for public policy making was also expressed by the respondents, who emphasized that such research must be practical rather than written “for the vacuum” or just to be eligible for PhD degrees.

Among the subordinate and affiliated institutions of the MES, the IMCE is the most closely connected to the NAESU. For example, in the Program of Collaboration between the MES and the NAESU, the IMCE is mentioned as the/a MES implementer for 72 activities out of 155 in 2017-20, and for 68 activities out of 85 in 2014-16 (as the Institute of Innovative Technologies and Content of Education). The UCEQA and IEA are barely mentioned: the current Program mentions UCEQA only once.

All the respondents from the IMCE spoke about close collaboration with NAESU institutions, both formal and informal. The respondents use the NAESU research findings, such as journal papers, monographs, methodological developments etc., mostly of methodological rather than scientific kind (this is the most typical for the field of secondary education). They also attend events organized by NAESU institutions, accompany the NAESU experiments, involve the researchers in their activities (for example, as teachers for experts who evaluate drafts of school textbooks), as well as in scientific and methodological commissions and in the expert evaluation of their regulations.

It should be noted that the IMCE’s and the NAESU’s functions, both formal and actual, are generally similar. From the respondents’ answers, one could conclude that the NAESU work which is the most relevant for the MES and the IMCE is the part in which the NAESU carries out the IMCE functions. The latter institution, according to its Statute, is supposed to provide scientific and scientific-methodological foundation for the modernization of the content of education by doing research and implementing its findings. However, in reality, the IMCE efforts are largely concentrated in the organizational-administrative plane, while the NAESU is an extension of this institution and takes over some of the work which is essentially the IMCE’s responsibility (primarily the research about teaching methods and implementation of those methods, the development of curricula and textbooks). Below, we list some of the similar functions of both institutions:

The UCEQA and IEA virtually never use the NAESU work. The respondents from the UCEQA noted that they had not used the NAESU work results as such (that is, its publications, events, etc.), but that they use the expert knowledge of particular academicians and employees of the NAESU institutions as they carry out their tasks, both current (determining the threshold grades for the External Independent Evaluation) and for particular projects, most of all for working on the PISA study in Ukraine. The NAESU researchers’ contribution to the preparation of the study was highly appreciated by the respondents, but the NAESU research findings they were familiar with were deemed rather theoretical, while the UCEQA needs are more practical. One of the respondents used a metaphor to describe it, saying, “We are two different worlds.”

The respondent from the IEA does not personally use the NAESU work, but his colleagues have worked with the materials of the TIMMS study, which was organized and whose findings were analyzed in Ukraine by the NAESU (however, the respondent noted that the report about the study has never been published). In addition, the IEA engages the NAESU expert knowledge by inviting NAESU representatives as working group members and by asking them to give expert conclusions about its research programs. The results of this cooperation were characterised as contradictory, partially positive and partially negative.

It should be noted that many respondents from the MES and its subordinate institutions, when talking about the collaboration with NAESU representatives by involving them in working groups, mostly mentioned the same names, emphasizing those individuals’ specific contributions to policy making. Therefore, we can assume that there is a relatively small (compared to the general number of academicians, associate members and employees of NAESU institutions) group of scientists within the NAESU whose expert knowledge is in the highest demand and whose contribution to policy making is the most significant.

As for the use of NAESU work in the activities of other central executive government bodies (the Ministry of Social Policy, the Ministry of Finance and the Ministry of Family, Youth and Sports), the in-depth interviews with representatives of the two latter institutions showed that they do not use the NAESU work, while the Ministry of Social Policy refused to be interviewed because it does not collaborate with the NAESU.

Use of the Results of the NAESU Research Activities in the Development of Legislation

In the period under consideration, two core laws in the field of the NAESU’s expertise were passed: the framework Law “On education” and the Law “On science and scientific-technological activities.”

Individual NAESU representatives were members of the working group that drafted the bill “On education,” and they represented the highest level in the Academy’s structure; that is, the working group included three Presidium members. They could have participated in the creation of the version of the bill which passed the first reading in the parliament. But it is impossible to isolate their contributions, except for the article on scientific and methodological support of education, which was supplemented before the first reading with a segment on the role and tasks of the NAESU. However, between the readings, organizations and institutions could submit their suggestions to the Parliamentary Committee on Education and Science, so at least we can analyze what the NAESU suggestions were about.

The NAESU institutions submitted a total of 71 suggestions and corrections to the draft bill before the second reading. 50 of them were made by the Institute of Higher Education, and the rest by six other institutions. A considerable fraction of the suggestions (10 out of 71) were about the definitions in Article 1, the law’s glossary. A few others reflected the NAESU positions on the standardization of education and conveyed an understanding of educational standards specifically as government standards. In addition, the Institute of Higher Education suggested to include a provision that would determine that the state provides educational services in the predefined amount and has to define the cost of these services for each student, but only on the level of general secondary education — there were no similar suggestions for the articles on other educational levels. In addition, the Institute of Higher Education offered its own version of the chapter on the financial-economical relations in education.

Five of the NAESU suggestions were accepted. The working group members decided to accept the definition of autonomy of educational institutions and the list of the levels of education which were suggested by the NAESU, and to accept the suggestion to reject an idea about a separate law on higher professional education (it should be noted that other stakeholders who also submitted suggestions for the same article opposed the creation of this separate law, too).

The NAESU suggestions for the draft bill “On science and scientific-technical activities” (10 in total) were exclusively about the status and rights of national sectoral academies of science (as well as the National Academy of Science of Ukraine). These were suggestions for Article 18 (National Sectoral Academies of Science) and Article 27 (Preparation of Scientific Staff and Their Further Training), as well as for the part of the Final and Transitional Provisions of the Law (paragraph 6, sub-paragraphs a-d) which were aimed to introduce changes in the Law “On higher education” according to the regulations of Article 27. The accepted suggestions were about the non-profit status of research institutions of the national sectoral academies of science, the coordination of these institutions’ activities by the Cabinet of Ministers without violating their autonomy, the possibility for the National and the national sectoral academies of science to found HEIs, as well as the possibility for their research institutions to train MAs, PhDs and Doctors of Science.

The absolute majority of the NAESU suggestions which were accepted in the final text of the bill were also submitted by the National Academy of Science and/or articulated in the alternative bills 2244a-2, 2244a-3, 2244a-4 or in the suggestions of other participants of the working group that finalized the government’s bill 2244a before the second reading.

Use of the Results of the NAESU Research Activities in the Work of Methodology Departments and Educational Institutions

According to the survey of the workers at regional institutions of the postgraduate pedagogical education system, the majority of them use the NAESU research findings in their work. 87% of the respondents participated in events organized by NAESU institutions (alone or in cooperation with other institutions) in 2014-17. Half of them participated in 1-3 events, and 30% in 4-6 events. 20 out of the 34 respondents who answered this question claim that these events are an important component of their work which improves the quality of their scientific and pedagogical activities, and 9 of them defined their role as an additional activity aimed to study a certain topic in-depth, which does not have any significant impact on their daily work. 77% of the respondents highly evaluated the relevance of the topics of the majority or a considerable percentage of these events. 70% of the respondents are definitely going to use the knowledge and skills they learned at these events in their work, and another 27% are probably going to do it. 70% have heard their colleagues speak about the NAESU events they attended, and these mentions were mostly positive in many or in nearly all cases. Judging from the responses we received, the experience of participation in the events organized by the NAESU is evaluated by IPPE workers as generally positive and useful.

92% of the respondents have read printed materials (papers, monographs) written by employees of NAESU institutions or in collaboration with them in 2014-17. 44% of them defined what they had read as very useful and confirmed that they were going to use the knowledge they had learned, another 42% of them defined it as potentially useful, and another 11% found only particular useful aspects in what they read. 80% of the respondents read NAESU journals in the specified period, and 84% of them said that the majority or a considerable fraction of the materials published in these journals were of high quality and were interesting.

In-depth interviews with IPPE managers showed that some of the studies produced by the NAESU lack a practical aspect, a possibility to use the new knowledge in the professional work (the studies in educational philosophy, history of pedagogy and comparative educational studies); meanwhile, some other NAESU works (textbooks, methodological recommendations) lack a systematic scientific approach, because they are based on their authors’ personal beliefs or experiences or on isolated experiments which need to be described.

In general, the respondents in the in-depth interviews were more critical of the NAESU’s research results and noted that they preferred foreign research in the field of education. In addition, they claimed that other IPPE employees do not read scientific papers because they do not have the time for it and because the language of these papers is a bit too exaggeratedly theoretical.

It is possible that this discrepancy in the evaluation of the use of NAESU research in the work of IPPEs is caused by the method of data collection (in the case of in-depth interviews, the respondents knew the interviewers personally and trusted them more), although another possible explanation is that it reflects the diversity of the respondents’ experiences.

In order to learn about the use of NAESU research findings in pedagogical higher education institutions, we conducted in-depth interviews with their representatives, as well as with several representatives of high schools who are involved in education policy making. All the respondents who work at HEIs knew about the NAESU and its activities and had different stories to tell about their interaction with the NAESU, including the use of the NAESU research findings. They had varying opinions about the NAESU. Some (including school teachers) noted that they did not use NAESU research results in their teaching or did not even know about them. Instead, they used foreign research.

Some respondents have negative opinions about some specific aspects of NAESU activities or of the collaboration with it:

“But what kind of interaction? It sometimes takes ridiculous forms. For example, at nearly all meetings of the Scientific Council, we approve thesis topics. Traditionally, the highest number of theses, the highest number of PhD students are in educational sciences. And it’s always funny for me that when the topic is announced, then the standard question always is whether the topic is coordinated. It is a unique case here in Ukraine when thesis topics must be coordinated with someone from the Academy of Educational Sciences; as far as I understand, it’s done so that studies are not duplicated. Because there are so many of them that there is a high risk that there will be many theses on the same topic, so they are coordinated, these titles. This is probably the only coordination I know about.”

Other respondents, on the contrary, have very positive opinions about the NAESU work and about the cooperation and possibility of cooperation with the Academy:

“As a teacher of courses in pedagogy, I want to say that we are very grateful to Vasyl Kremen for this encyclopaedia of education which we use, which the students use, which we have in the library. I must say that we feel considerable support through the textbooks… I can say with much appreciation about the textbook in didactics, published by Volodymyr Bondar, Svitlana Sysoyeva and Yevtukh. These are works whose target audience are professors of courses in pedagogy.”

Some respondents also had some experience of working with the NAESU by publishing their papers in the NAESU’s journals. One of them shared the following opinion:

“I write a paper and send it to them. They reply the next day, “Your paper has been accepted.” There were never any questions or clarifications from reviewers, nothing. I mean, they pretend they review something. Maybe they do. But you understand that real reviews must be anonymous, there must be a system of communication between authors. But it doesn’t happen. They write that it will be reviewed and might be rejected. I think that someone there takes a look at these articles, so that something completely terrible doesn’t pass. But this is not real peer review. And external experts, as far as I understand, are not involved there, because all of this is supposed to take time, there should be a test for plagiarism and so on.”

Some NAESU representatives simultaneously work at pedagogical universities. The respondents who have such colleagues talked about constant collaboration with the NAESU:

“We have systematic visits by representatives of the Academy of Educational Sciences, we have direct information, because two of our representatives are members. For example, the previous rector of the university was a member of the Presidium…. So everything that happened at the Academy of Educational Sciences and all the decisions by the Presidium, all the research results, we received direct information about them at our meetings of Scientific Councils, Rectorates and so on.”

Recommendations for the National Academy of Educational Sciences of Ukraine

We suggest that the NAESU reconsiders its procedures for organizing competitions for the funding of research projects and its procedures of quality assurance of studies, ensures mandatory participation of external reviewers (unaffiliated with the NAESU) in these procedures, and tries to avoid any conflict of interest.

We recommend that NAESU separates its areas of responsibility (formally and in practice) from those of the IMCE, leaving the tasks of updating the content of education, creating curricula and textbooks to the IMCE in order to avoid functional overlap, and that the NAESU focuses more on the scientific research as such.

We recommend the NAESU to pay special attention to the external communication of its research findings and the work of its press center, because the better these findings are presented in the public space, the higher is the outreach to the consumers and the visibility of the institution itself.

We advise the NAESU to change the editorial policies of the NAESU’s own journals, bringing them up to the global standards.

We recommend that the NAESU introduces formal criteria of evaluation of its employees’ scientific papers, such as the citation index and publications in journals which are indexed by Scopus or the Web of Science.

Recommendations for the National Ukrainian Council on the Development of Science and Technology

We recommend the Council to define the priority topics in education which need to be researched.

We recommend the Council to initiate a thorough audit of the NAESU’s scientific activities in terms of the relevance and novelty of its research in the context of the needs of Ukrainian education and the global tendencies in educational research.

We recommend to review the rules for the distribution of funding for research, in order to take into account the priorities, the relevance and the scientific quality.

We also recommend to introduce grant-based funding as the prevalent way of funding scientific studies, which would open the opportunity to receive the funding to conduct these studies for all research institutions, regardless of whether they are public or private, and for all teams of authors who can prove their capacity to provide the conditions for the effective use of the funding. We also recommend to involve representatives of stakeholders, especially the MES, in the competition commission that would consider the grant applications.

Recommendations for the Ministry of Education and Science of Ukraine

We recommend to allocate resources for the development of the Ministry’s subordinate institutions which will productively work directly on the tasks of updating the content of education and providing scientific and methodological support.

We recommend to introduce long-term planning of priorities in educational research. We also recommend the MES to announce open competitions of research projects and policy analyses in education, in order to use the budget money to obtain products of higher quality, which can be used in fulfilling the Ministry’s functions.

In order to improve the quality of educational policies, we recommend that the MES changes the processes of policy making and the development of regulations, introducing the stage of data collection and analysis, reviewing the relevant research, and forming the requests for the data and studies which need to be collected and carried out.

Recommendations for the Ministry of Finance of Ukraine

When determining the amount of funding for the NAESU, we recommend the Ministry of Finance:

• to take into account the official position of the MES about the concrete practical use of any particular NAESU research project for the policies of the Ministry of Education and Science; in particular, to request from the MES the list of the specific steps needed to implement certain policies and individual regulations in which the NAESU research findings could be used;

• to take into account the evaluation of the NAESU’s report for the previous period, made by the National Ukrainian Council for Science and Technology.