A review of policy and its implementation - lessons drawn from recent audits Algemene Rekenkamer Lange Voorhout 8 P.O. Box 20015 2500 EA The Hague THE NETHERLANDS A review of policy and its implementation lessons drawn from recent audits Contents 1 A Review of policy and its implementation 1 1.1 Introduction 1 1.2 Policy and its implementation 2 1.3 Has the policy been fully implemented? 4 1.4 Are policies being implemented properly? 6 1.5 Concluding remarks 9 1.6 Response by the Minister of Finance on behalf of the Prime Minister and the government 1.7 Court of Audit afterword 10 14 A review of policy and its implementation - lessons drawn from recent audits 1 A Review of policy and its implementation 1.1 Introduction The budgets for 2003 give the impression of an ambitious government aiming for new policies in fields such as law and order, care, the integration of ethnic minorities, and education. But do these policy announcements actually have any impact in practice? In order to answer this question, the Netherlands Court of Audit has selected a number of major themes and compared the situation described in its audit reports of the past few years with the situation as at the end of 2002. The Court’s main concern was to ascertain whether the problems raised at the time have now been solved. This memorandum sets out the findings of this review. The themes examined were as follows: • Law and order • The repatriation of rejected asylum seekers • The integration of and Dutch-language classes for ethnic minorities • Food safety • Problem youngsters • Early school-leavers • The reintegration of invalidity benefit claimants • Information in the care sector • Infrastructure • Climate policy • Tax as a policy instrument • Peace operations Many of these themes also have high priority in the Court’s audit strategy.1 The main reason for selecting these particular themes was the fact that they are areas in which central government plays a vital role, particularly for vulnerable groups in society. 1 The Court’s priority areas are: the environment, education, care, income, housing and law and order. 1 A review of policy and its implementation - lessons drawn from recent audits 1.2 Policy and its implementation The Court of Audit is not concerned with the substance of government policy: this is the preserve of politicians. The Court’s task is to assess whether, in the light of the decisions made, policy is clear and has been properly implemented. The main concern is whether policy is clearly targeted at a particular issue, what the scale of the problem is, what the target group is, what sort of solution the policy is intended to bring about, and when. When it comes to implementation, the Court considers questions such as: • What activities have been planned in order to achieve the policy objectives? • Have the planned activities been performed, and have they been performed efficiently, i.e. at the lowest possible cost? • Have the policy objectives been achieved, i.e. has the problem been solved or has the situation improved? • Are any improvements due to government policy or to developments that are not connected with government policy? The Court primarily uses information supplied by ministries. This makes sense: after all, the ministries need this information themselves both to implement their policies and to report on their performance. They are also obliged to evaluate their policies every five years. In other words, this information should theoretically answer all the Court’s questions. However, it is clear from the results of the audits performed by the Court that this is not the situation in practice. Only in very few instances is all the information actually available. The Court regards this as a crucial point: only if good-quality information is available both on policies and on their implementation and impact can the minister and/or the House of Representatives take well-founded decisions on whether to continue with a policy, discontinue it or modify it. The VBTB process (involving a shift in emphasis from input-oriented management to results-based management) was introduced a number of years ago and is intended to result in a new, transparent method of budgeting for and reporting on government policy. It requires ministers to demonstrate what their policies were intended to achieve, how they intended to achieve it, how much money was set aside for the purpose, and at the end of the day, what they actually achieved, and at what cost. However, improvements in the flow of policy information have been slow in coming. 2 A review of policy and its implementation - lessons drawn from recent audits One of the basic problems is that many ministries do not possess any information on the degree to which their activities have actually succeeded in achieving certain policy objectives. Because such information is not available, it is difficult to say which aspects of policy have been implemented and which have not. Moreover, the absence of this information inevitably means that it is not possible to assess the decision to implement only certain aspects of policy. Where information is available, it often relates solely to policy that is in the process of implementation and its results. However, this information is also often inadequate, which makes it difficult to describe with any degree of accuracy the quality of policy implementation and the results of activities. In most cases, what information is available leads to the conclusion that the policy in question has not been properly implemented. The situation may be represented diagrammatically as follows: Has the policy been fully Has the policy been properly implemented? I implemented? II No policy It is not clear whether the policy It is not clear whether the policy information has been fully implemented has been properly implemented Policy The policy has The policy has The policy has The policy has information been fully not been fully been properly not been available B implemented 1 implemented 2 implemented properly available A implemented 2 Although the answers to these questions (assuming that they can indeed be answered) vary from one audit to another, it is hardly ever the case that all policy information is available and that the Court may conclude that the policy in question has been implemented both fully and properly. The Court of Audit wishes to emphasise this point, because it implies that: • in many cases, the problems the policy was intended to solve persist for longer than was intended when it was decided to pursue the policy in the first place; • the House of Representatives may well be unaware of this, because no information is available on certain aspects of policy implementation. 3 A review of policy and its implementation - lessons drawn from recent audits In order to illustrate these points, the Court has taken a number of examples from recent audits. For more detailed information, please see the relevant sections on each theme. 1.3 Has the policy been fully implemented? It is not clear whether the policy has been fully implemented (IA) The absence of relevant information means that it is not always clear whether the policy in question has been implemented and, if so, whether it has been implemented in full. An example of this is the scheme for bringing early school-leavers back into education. An audit conducted by the Court at the end of 2001 revealed that the records were of such poor quality that it was not possible to calculate the precise number of school dropouts. This means that neither the minister nor the House of Representatives can know to what extent the policy aimed at this particular target group is being implemented. The Court’s own audit showed that the scheme had failed to reach around 13% of all registered dropouts. The policy on shelters in the community is another example of an area where no hard facts are available on a particular problem group – in this case homeless young people. No one knows exactly how many homeless young people there are. The Minister of Health, Welfare and Sport does not have a policy on individual target groups, which means, for example, that no statistics are kept on this category of young people. To compound matters, the various experts and care organisations are unable to agree on the precise definition of a ‘homeless young person’. Moreover, no data are available on whether or not the services currently on offer are adequate. An investigation carried out by the Court in 2002 showed that there were at least 2,700 homeless young people in the Netherlands. The policy has not been fully implemented (IB2) In cases where there is a reasonable amount of policy information, the available data sometimes show that the policy in question has not been implemented in full. This may be because it is simply not feasible to implement all aspects of the policy. A good example of this is the work of the inspectorates. An audit performed in 2002 revealed that the inspectorates were not discharging all their statutory duties. Their statutory responsibilities are so wide-ranging that they are forced to 4 A review of policy and its implementation - lessons drawn from recent audits make choices. However, there is no clear rationale for these choices. In other words, the inspectorates are handling only some of the problems they are supposed to handle. This poses a direct threat to the standard of enforcement. A 2002 audit of measures to combat juvenile crime showed that the files on over 40% of first-time young offenders contained no information on any action taken. And yet the offences involved were no different or less serious than the cases where the police had taken action. Another example is the practice of releasing suspects due to a shortage of prison cells. An audit performed in 1997 showed that some suspects were being released from pre-trial detention. Following an examination of the files, the Court of Audit concluded that there was a substantial risk that the sentences subsequently imposed on some of those released (whether custodial sentences or alternative sanctions) would never actually be enforced. In the meantime, the problem has been largely resolved by the practice of releasing prisoners early or temporarily whenever there is a risk of a cell shortage. The Court concludes that the problem of the release of suspects has now been transmuted into a growing problem of the release of convicted offenders. The consequence is that it remains unclear whether the policy on criminal law enforcement is being fully implemented. In many cases, it is difficult to explain exactly why policy has not been fully implemented. An audit conducted in 2001 into efforts to combat educational disadvantage showed that half of all plans made by local authorities to tackle this problem did not take account of their statutory obligations. When the new Sheltered Employment Act came into effect, many local authorities failed to fulfil their statutory duty to ascertain what had happened to some or all of those who had been removed from the waiting lists for sheltered employment when the new criteria took effect. Overall impression as to whether government policy is being implemented in full (I) Not a single audit performed by the Court of Audit in recent years has shown evidence of a policy that has been fully implemented. Either insufficient information was available to warrant such a conclusion, or the information that was available suggested that the policy had not been fully implemented. 5 A review of policy and its implementation - lessons drawn from recent audits The Court cannot avoid the impression that the civil service has far too many responsibilities and may even have been struggling to keep its head above water for many decades now. In order to survive, the civil service has decided to do what it can instead of what it should. 1.4 Are policies being implemented properly? It is not clear whether the policy has been properly implemented (IIA) Here too, reliable policy information needs to be available in order to detect any shortcomings in policy implementation. Without any information on the process and the results, it is impossible to form a judgement on the quality of policy implementation. The audits performed by the Court in recent years have thrown up many examples of situations where there is simply no information on the implementation of government policies. In most of these cases, the missing information related to the impact of the policy. We should now like to give a few examples. When the policy aimed at combating educational disadvantage was audited in 2001, the findings made clear that the government did not know whether the policy was actually having any effect, despite the availability of research data in this field. Although the education gap has not narrowed during the past 20 years, nobody knows whether it would have widened had no action been taken. It is unclear whether many of the tax incentives introduced during the past decade have actually had their intended effect. With regard to asylum policy, the government does not know whether the majority of unsuccessful asylum seekers have actually left the Netherlands, despite the fact that its policy is aimed explicitly at encouraging them to arrange their own return. For a long time, it was unclear whether the policy of increasing the capacity of the country’s police forces had been properly implemented, because the figures on the size of the forces were not reliable. Improvements have now been made in this area. The Court of Audit also encountered other examples in which clear improvements had been achieved in relation to policy information. The Ministry of Defence, for example, has structured the flow of information 6 A review of policy and its implementation - lessons drawn from recent audits on peace operations in such a way that key lessons learned from one operation can now serve as input for subsequent operations. The Ministry of Transport, Public Works and Water Management has developed a standard decision-making procedure for infrastructural projects in which information plays a key role. The policy has not been properly implemented (IIB2) In some cases in which the relevant minister does collect information, the information shows that the policy in question has not been properly implemented. It is clear from the practical examples that all sorts of different factors may be at work here. Conflicting policy objectives The Ministry of Economic Affairs is responsible for certain aspects of the government’s climate policy, the aim of which is to reduce CO2 emissions. At the same time, the prime role of the Ministry of Economic Affairs is to foster the country’s economic growth, and measures to stimulate economic growth can easily raise CO2 emissions. The government’s climate policy does not suggest ways and mains of dealing with such conflicts, which need priorities to be set on a government-wide basis. As a result, it is not clear whether the Netherlands will be able to achieve the CO2 emission targets set for 2010 under the Kyoto Protocol. Another example concerns the capacity of the Dutch police forces. Some of the progress made in increasing capacity was offset by the terms of the collective agreement reached in 1998, under which the working week for police officers was reduced from 38 to 36 hours. This reduced the available police capacity by 5% at a stroke. Insufficient thought given to policy decisions Giving insufficient thought to policy decisions raises the risk that the policy in question will not be properly implemented. This is illustrated by the prison construction programme. A report published in 1999 claimed that most of the practical problems surrounding the programme were caused by the political and social clamour for a rapid increase in the number of cells. The government’s current programme on law and order adds to the pressure for an expansion in prison capacity. Insufficient account has been taken of its effects in financial, staffing and legislative terms. The new government has been left to decide whether the current measures designed to make more efficient use of the existing capacity are indeed adequate. When the time comes to take this decision, there is 7 A review of policy and its implementation - lessons drawn from recent audits a risk that the programme will again be encountering the type of practical problems highlighted by the Court of Audit. There will also be a greater security risk if (as is already happening at the moment) prison officers need to be recruited on temporary contracts, without any prior screening, in order to be sure that new prisons are properly staffed. Lack of rigorous management by the ministries In some cases, the implementation of government policy is difficult to manage because authorities other than central government are responsible for putting it into effect, often in conjunction with civilsociety organisations. This may result in a fragmented approach being taken to complex problems, such as those of juvenile delinquency and law and order in general. Where the successful solution of a problem hinges on effective cooperation between executive agencies and government bodies at various levels, the ministries concerned should not hesitate to retain a firm grip on policy implementation. Many policies suffer badly from an absence of central authority. The way in which policies are funded may also prevent them from being implemented in a cohesive fashion. One example of this is the way in which programmes for integrating immigrants, both newcomers and those who have lived in the Netherlands for some time, are funded. The strict separation of financial flows prevents the development of customised programmes. Reorganisations Often, one of the reasons for reorganising government departments is to improve the quality of policy implementation. In practice, however, such improvements often take a long time to make themselves felt. One example of this is the merger of the National Commodity Inspectorate and the Government Meat and Livestock Inspectorate, which were combined in July 2002 to form a new agency known as the Dutch Food and Non-Food Authority, with the aim of improving the standard of food safety checks. An audit performed by the Court of Audit revealed that the new authority was not performing well. As a result, the authority has so far failed to achieve any improvements. In the social security sector, it will undoubtedly be several years before the body responsible for implementing employee insurance schemes (founded on 1 January 2002) will be operating according to plan. The 8 A review of policy and its implementation - lessons drawn from recent audits new body is the result of the amalgamation of five formerly separate benefits agencies. Intractable problems As we have already mentioned, information is a problem in education. The ministry is aware of the problem and has taken steps to improve the situation. The ministry is currently preparing for the introduction of an ‘education number’, a unique registration number that will be allotted to every single school pupil for use in all records. Unfortunately, the system will not become fully operational until 2006, due to legal complications and technical problems. Overall impression regarding the quality of policy implementation (II) There have been few occasions in recent years when the Court of Audit has been the bearer of good news. It would be facile to attribute this solely to its selection of subjects for audit. If this were true, the theoretical implication would be that all policies not investigated by the Court had been both fully and properly implemented. This is highly unlikely, however, given the consistency with which the Court identifies practical problems and gaps in the information on policy implementation and impact. Indeed, the Court is firmly convinced that the government has bitten off more than it can chew. 1.5 Concluding remarks The officials responsible for the implementation of government policy are not generally to blame for the fact that the Court is forced to reach the same alarming conclusions about the implementation of policies and the flow of information in all kinds of fields. The Court believes rather that the recurring problems affecting the implementation of policy are caused by: • inadequate preparation and overambitious planning; • the overrating of policymaking and the underrating of policy implementation; • an insufficient interest in reporting, evaluation and costing; • an apparent inability to learn from the past. While no hard data are available on policy implementation, it is clear that something must be seriously wrong if a review of the audits performed over the past four years reveals so many examples not only of policies 9 A review of policy and its implementation - lessons drawn from recent audits that have not been fully or properly implemented, but also of policies of which we have no idea whether they have been either fully or properly implemented. The Court is particularly concerned about policies that have not been fully implemented (i.e. where they have affected only a certain proportion of the target group, where not all the proposed measures have been taken, or where there are gaps in the requisite records). It is impossible to decide whether a policy has been properly implemented if certain aspects of it have not been implemented at all. Moreover, published reports often pay scant attention to such policies, with the result that nobody realises that the government has simply failed to implement certain aspects of its policies, without explaining why. It is apparently impossible to strike the right balance between policy, human and financial resources, and the amount of time available for implementation. The Court regards this apparent incompatibility – gap, if you like – between political goals and their practical implementation as one of the greatest threats to the credibility of central government, and to public confidence in the government. It also feels it has a duty to focus on this problem in future audits, and expresses the hope that parliament will seek to narrow the gap between political goals and the reality on the ground. 1.6 Response by the Minister of Finance on behalf of the Prime Minister and the government On 17 March 2003, the Minister of Finance responded to this general review on behalf of the Prime Minister and the other members of the government. The main points of his response are reproduced below. Main points of response The government points out first of all that many of the social problems for which people look to the government for a solution are both highly complex and intractable. This means that the government becomes something of a repository for problems which the ‘open market’ is unable to resolve. Political aspirations expressed in the past may sometimes have created the impression that the government is capable of solving such problems. Yet the government is no panacea. The government acknowledges that a realistic view of its aims and performance can help to enhance its credibility. It believes that the VBTB operation should help to improve the transparency and realism of the government’s policy 10 A review of policy and its implementation - lessons drawn from recent audits objectives. The government believes that the main conclusion drawn by the Court is that information about the implementation of policy is often missing and that, where such information is available, it is clear that the policies in question have not been properly implemented, due to a number of factors. The Minister writes that the government is aware of the importance of implementing policy as fully as possible, and of sufficient information being made available to allow it to ascertain whether policies have indeed been fully implemented and/or to take any remedial action that may be needed. He refers to the general measures that have been taken to improve the flow of information and the response to policy implementation, for example in relation to bodies with a statutory task, and as part of the VBTB operation. In this sense, a systematic approach to budgeting and reporting can prompt more methodical organisation of information on policy, policy implementation and the impact of policy. It is quite simply a fact of life that the best way of assessing and learning is by looking and comparing, and ‘fixed’ benchmarks are needed for this. Society cannot be moulded into a pre-ordained shape, although it is true that social trends can be controlled to a certain extent. The VBTB operation is designed to communicate the government’s objectives and show people how it has reached a particular decision in a particular context. The social and political context is such that it is not always possible to think through all the potential implications of a policy in advance. • While there will always be conflicts between different policy objectives, the government is seeking to improve the transparency of its decisions in such cases. • Because the context is constantly changing, policy can never be 100% effective. • Information provided on policy shaped and/or implemented locally should be adequate for local democratic control, and proportionate to the goal the policy is intended to achieve. • To inspire confidence, government needs to bridge the gap between its aspirations and its actual implementation capacity. Confidence springs from clarity about the responsibilities of government and what it can realistically achieve. • Relevance to policy management, information value and the relationship between cost and benefits are key factors in deciding whether or not to organise the flow of information. 11 A review of policy and its implementation - lessons drawn from recent audits The VBTB operation The VBTB operation (the Dutch acronym stands for ‘From Policy Budgets to Policy Accountability’) will reveal how realistic government targets are, where coordination problems occur and where the division of responsibilities is unclear. The government needs to be aware of the risk of raising hopes too high. Public confidence in the government springs from clarity about its responsibilities, and about what it can realistically achieve. Many of the targets the government set itself in the past were clearly overambitious. As regards the provision of information on policy shaped and/or implemented on a local basis, VBTB policy is that ministers should also be expected to report on results for which they are not directly responsible, but which do give an indication of the effectiveness of the policy framework for which they are responsible, and which may be highly relevant to policymaking. In over half of such cases, direct and indirect responsibilities have been identified and the ministries are now doing their best to ensure that their performance is reflected in both budgets and reports. The government regards the findings in the draft report entitled Between Policy and Implementation as valuable support for its efforts to further improve the development of policy and the flow of policy information, and how this is reflected in the VBTB budgets and annual reports. However, it feels that the suggestion that ‘only if good-quality information is available both on policies and on their implementation and impact can the minister and/or the House of Representatives take wellfounded decisions on whether to continue with a policy, discontinue it or modify it’ is too radical a conclusion. Although it is undoubtedly true that decision-making benefits from good information on policy matters, other factors are also important. There may for instance be a social – and hence political – need to take rapid action, without any in-depth problem and policy analysis in advance. Obviously, as the situation unfolds, so the government needs to take stock of its policy, decide how it should be implemented and figure out the type of information that needs to be provided about it. Its aim in doing so should be to underpin its decisionmaking as firmly as possible. Effective reporting procedures The government believes that ministers can effectively use the budgeting and reporting procedures to report on the progress they are making in 12 A review of policy and its implementation - lessons drawn from recent audits drawing lessons from the past. Where these lessons involve a number of policy areas and ministries, the government suggests that a decision should be taken each year as to whether they should be included in the central government annual financial report, so as to maximise the learning effect throughout the public sector. Whilst the minister acknowledges that the Court of Audit’s conclusions merit a ‘meta report’ of their own, he points out that it is wise to avoid too many reports on reports, and responses to responses, and the inevitable coordination problems this leads to. Learning opportunities So as not to raise any false hopes, one also has to be clear about how much it is possible to learn from policy. In the case of policy information, it is important to make a distinction between the intended effects of policy on citizens (i.e. the policy outcome) and the actual results of policy implementation (i.e. the policy output). As the minister sees it, Between Policy and Implementation examines matters mainly from the latter perspective. The minister wishes to stress the difficulty of obtaining information purely about policy outcomes. After all, it is difficult to prove that it was not factors other than the policy in question that played a predominant role in achieving the intended effect. Whilst exogenous factors may be involved (for example, economic growth, labour market trends or the behaviour of the target group), the role played by third parties in the implementation of policy may also help to explain why the intended effects are not always achieved. Although policy evaluations always generate information that can be used to refine government policy, they are always expensive undertakings. The minister believes that whether or not information should be structured depends primarily on its relevance to policy management, its value to the House of Representatives and a costbenefit analysis. Government proposals in response to ‘Between Policy and Implementation’ The government regards the VBTB operation as a general attempt to resolve the shortcomings identified by the Court of Audit in information on government policy and its implementation. Although the benefits will take some time to make themselves felt, initial results suggest that the approach is both realistic and effective. The government therefore proposes that: 13 A review of policy and its implementation - lessons drawn from recent audits 1 the VBTB operation should be pursued with full vigour, as it should enable the government to: • assess policy objectives, resources (including the resources deployed by and the costs incurred by executive agencies) and results in the light of realistic targets and risks; • build up a transparent picture of a policy setting that is virtually by definition complex and intractable; • report systematically on its policy plans, the implementation of its policies and their impact; • to learn in a systematic way from the past by conducting policy reviews and adapting future policy accordingly. 2 starting with the Budget Memorandum for 2004, specific and realistic targets should be formulated for the priorities set by the government. These targets should be based on budgets and should reflect the intended results of policies and their implementation. 3 a decision should be taken every year as to whether certain lessons learned from the Court of Audit’s conclusions and recommendations in the field of results-based management should be included in the central government annual financial report, so as to further their assimilation throughout the public sector. 1.7 Court of Audit afterword In his response on behalf of the Prime Minister and the government, the Minister of Finance writes mainly about the provision of information on the implementation of government policy, and points both to the opportunities created by the VBTB operation and to its limitations. The Court of Audit agrees that the provision of information is indeed a crucial issue, and wishes to respond below to the comments made by the Minister of Finance. The Court of Audit would like to point first of all to the apparent incompatibility that exists between the government’s political ambitions and the day-to-day practice of policy implementation. It is this gap between theory and practice that the Court of Audit sees as the core of the problem. It constitutes one of the greatest threats to the government’s credibility, in that the problems policy is supposed to resolve may either remain unresolved or take longer to resolve than planned because the policy either is not properly implemented or is not implemented at all. 14 A review of policy and its implementation - lessons drawn from recent audits In this respect the Court of Audit agrees with the government that the VBTB operation has prioritised the formulation of clear objectives accompanied by specific targets. Ideally, the presence of a mechanism for reporting on past achievements should give a clear indication of how realistic planned policy is likely to be. The Court of Audit therefore regards the VBTB operation as a vital disciplinary tool that should give both ministries and the House of Representatives an idea of the gap between policy objectives and results, and the practicability of policy. Although the Court therefore welcomes the government’s plans vigorously to pursue the VBTB operation, it feels it is important to bear in mind that, whilst the operation should ideally improve the flow of information, it will not automatically close the gap between policy ambitions and implementation. Further to the government’s comments about the VBTB operation, the Court of Audit wishes to emphasise that it, too, believes it is important to carefully weigh up the cost of data collection against the benefits derived from it in terms of managing policy and informing the House of Representatives. Indeed, it is this very point that is addressed by the Order on Performance Data and Evaluations in Central Government. At the same time, the Court of Audit believes that decisions to pursue or change certain policies should be based as far as possible on reliable data. The fact that other factors, such as a political or social need for rapid action, may play a role should not detract from this general principle. The Court of Audit welcomes the government’s plans to include in the central government annual financial report certain conclusions drawn from audits performed by the Court of Audit that apply throughout the public sector. Finally, the Court of Audit wishes to make clear that, although it has no intention to publish an endless series of audit findings, it wishes to perform more systematic ‘meta evaluations’ of previous audits, so as to identify long-term trends. 15
© Copyright 2026 Paperzz