Association of Local Government Auditors

  • narrow screen resolution
  • wide screen resolution
  • Decrease font size
  • Default font size
  • Increase font size
  • default color
  • red color
  • green color

 

Alternate title

Reporting on the Performance of Government - March 1992 | Print |
Written by Harry P. Hatry   
Editor's Note: This article is adopted from a presentation Mr. Hatry made in May 1991 to the Mid-America Intergovernmental Audit Forum. We are grateful to Mr. Hatry and to the Forum for allowing us to include it in the Local Government Auditor's Newsletter.

Government is for people, and people are the government's customers. A key issue for government, for which its officials should be held accountable, is with what quality the government delivers services to its customers and what results are achieved.
Each program and agency should be held accountable for service quality, not just that the expenditure complied with the law. Quality has three important dimensions, each of which should be measured. Basically, the service should (1) produce beneficial, positive results, (2) avoid major negative consequences, and (3) be delivered in a way that is timely, helpful and fair.

In the 90's, a number of events have occurred that have led to an increased interest in the topic of performance measurement and evaluation.

  1. Quality in the private sector has been affected by Japanese competition; a country that was known for inferior products suddenly became known as the quality producer.
  2. Management publications such as Peters' and Waterman's In Search of Excellence and those of Edwards Deming have emphasized the needs for service to clients, knowing the customer, and providing quality service to clients. Public officials have come to feel that these same principles apply to public services as well.
  3. A more recent development, Total Quality Management (TQM), emphasizes the idea that every employee of a public sector agency should be continually involved with trying to improve its product quality. This implies the need for measuring that quality so public employees will know how they are doing and how successful they have been in their attempts to improve it.
  4. The 1990 Chief Financial Officer's Act emphasizes that each federal agency is responsible for the systematic measurement of non-financial as well as financial performance.
  5. Proposed Senate Bill S20, the Federal Program Performance Standards and Goals Act of 1991, requires each agency to remove service results, set targets, and report on achievement of these goals each year.
  6. Some professional associations have begun to develop performance indicators for their own professions.
  7. Many states are requiring formal performance reporting by school districts in their states.
  8. Inspectors general for some major federal agencies, such as Health and Human Services, have introduced outcomes evaluation into their work.
  9. At the state and local level, the Governmental Accounting Standards Board (GASB) has undertaken a major effort focusing on service quality and efficiency in twelve functional areas. Those areas include fire and police protection, solid waste handling, water supply and waste water treatment, road maintenance, public transit, public health and hospitals, public assistance processing, economic development, elementary and secondary education, and higher education. (The first reports, including the overview report and reports on elementary and secondary education, public transit, water and wastewater, public health, and solid waste are currently available from GASB.)
The National State Auditor's Association passed a resolution last year urging GASB to give its project very high priority in terms of publicity, visibility and staffing.

PERFORMANCE INDICATORS
Agencies, auditors and public officials need to make distinctions between the immediate physical work of an agency and the outcomes of that work. For example, human service programs typically report the number of clients they have seen, but much more important is the number of clients who actually improved over the condition they had when they came in.

Indicators of efficiency (productivity) when computed in the public sector traditionally are computed by dividing the number of physical outputs by inputs. GASB, however, has added a new and important twist by using outcomes in the numerator. For example, in human services, programs should calculate not only the number of clients seen per staff hour but also the number of clients who achieve a certain minimum level of improvement after receiving services (per staff hour). Public agency work should have to meet tests of quality before it should be counted as real product when calculating efficiency and productivity.

GASB reports recommended that agencies routinely be encouraged to provide explanatory information with outcome and efficiency data. Agency managers should be given the opportunity to provide explanatory that information in whatever form they want.

ROLES FOR PERFORMANCE AUDITORS AND INSPECTOR GENERAL OFFICES
b Auditors and inspector general staff have at least three roles. Those roles are to:

  1. Periodically perform audits of the quality and outcomes of major programs. This includes collecting new data on outcomes and quality. (Currently, however, some agencies do not have the legal authority to evaluate service quality.)
  2. Audit and check on the quality and accuracy of data reported by service agencies relating to service quality and outcomes.
  3. Assess the extent to which each operating agency measure service outcomes and quality on a regular basis.
In each role, auditors should provide recommendations for improvement that emerge from their audit findings.

PERFORMANCE MEASUREMENT PROCEDURES
There are at least three basic ways to collect data for measuring performance: using agency records, trained observer ratings and customer surveys.

  1. Agency records are most frequently used for collecting data regarding agency performance. Examples of data collected by using agency records include such information as response times, complaints, recidivism in human service programs, mortality/morbidity statistics, and employment data.
  2. Trained observer ratings, performed by inspectors and other trained personnel using rating scales, are generally used for assessing the physical condition of facilities such as buildings, roads, parks and housing, and for food service and environmental inspections; but they also have been used by human service agencies to assess improvement in clients.
  3. Customer surveys are used to obtain ratings from customers about various service characteristics and the extent to which they have helped the customers. Customer surveys can be used to (a) obtain actual data not readily available in program records about a number of specific service characteristics; (b) obtain ratings of various service characteristics; (c) identify customer concerns; and (d) obtain recommendations for improvements from the clients. The selection of service characteristics should come from program managers and customers, as well as elected officials. Mail surveys can be very useful to auditors and IG's for collecting performance data. While inexpensive, mail surveys can have major validity problems because of low response rates. But, mailings from government to program clients are likely to have higher response rates than surveys of the general population. Use of multiple mailings and telephone follow-ups can substantially increase response rates.
In addition to these three, there are other ways that data can be collected.

Interviews with experts and focus group meetings are becoming popular techniques among auditors and evaluators; both of these are qualitative approaches. In the first phase of audits and inspections, interviews and focus group meetings can be used to help identify what needs to be measured. At the end of audits and inspections, these can help interpret the information you obtained, what happened, and what might be done to improve it.

In another procedure, "customer role playing", audit and inspector general staff play the role of customers to assess the quality of programs' responses, such as for evaluating equal opportunity programs and agency responses to client's complaints. This procedure is intriguing but is likely to require considerable staff effort to generate statistically useful data. Thus, such procedures have only limited applicability.

DATA REPORTING WARNING: AVOID OVERLY AGGREGATED DATA
One of the major dangers in data analysis is "aggregation-itus". If you only report aggregated data, you are highly likely to mislead everybody. Appropriate breakouts of performance data will greatly improve the usefulness of the data.

The importance of presenting key breakouts of performance data in addition to overall aggregative data in your performance reporting needs to be emphasized. Vital information is often contained in breakouts. For many, if not most, programs it will be highly useful and informative to break out performance indicator data by: organization unit, facility, and geographic area. Breaking out success rates for groups of work (e.g., clients) for each major category of work difficulty is important to provide an adequate perspective on program accomplishments and to be fair to program staff.

For example, for public programs (federal, state, or local) aimed at helping economic development through assistance to businesses, performance data are likely to show some important differences among regions, size of business, and the industry type. To understand what is going on and to do a proper audit will require diving into such detail and performing some statistical analysis to access the interrelationships among such variables.


HOW TO TELL WHETHER PERFORMANCE LEVELS ARE GOOD OR BAD
A number of comparisons can be made to make such assessments.

  1. Sometimes, but rarely, standards exist for assessing whether the measured performance levels are good or bad.
  2. All agencies compare current performance to past performance. Previous years indicator values are usually a useful way to determine whether or not performance is improving or whether conditions are worsening.
  3. Breaking out the data by different organizational units, client groups, or geographical areas, will let you compare each's performance to determine where the service seems to be performing well or not well.
  4. One of the favorite bench marks of the press is performance of other jurisdictions. The trouble with this is that few jurisdictions currently are sufficiently comparable and can provide comparable performance data. Federal government agencies sometimes require standard data collection procedures. In this case, state and local agencies can compare their performance against other states.
  5. State and local offices can sometimes compare their performance to that of other offices--if there are different offices, facilities, districts, etc., performing similar work for similar types of clients. (Agencies making such comparisons should be concerned about differences in client difficulty and differences in services provided, so, agencies need to be careful in interpreting across jurisdiction performance.)
  6. Occasionally, an agency can compare its own performance against the private sector.
  7. Finally, if the agency sets targets for itself, actual achievement can be compared against those targets.
CONCLUSION
All of the aforementioned procedures have been used by one or more government agencies; but, by and large, use of some of these procedures are still rare. Data processing technology, with microcomputers, has been a major breakthrough, permitting rapid tabulation and reporting of performance data.

Where many technical problems still exist, substantial technical improvements in measuring the quality and outcomes of most federal, state, and local programs currently exist. Auditors of public programs should be in the forefront in use of these improvements.

Harry P. Hatry is the Director of the State and Local Government Research Program at the Urban Institute
 

From the Quarterly Editors


  • Anytime a group of auditors gathers to discuss the American Recovery and Reinvestment Act (ARRA), words such as "unprecedented" and "new paradigm" often follow. The idea of covering such a groundbreaking topic seemed overwhelming at first. 

    Read more...


  • Congratulations to all those who worked hard to put on an incredible 2010 ALGA Conference in San Antonio, Texas.  Even with a dismal economy, many of you saw the value in enhancing your audit skills, networking, and supporting our only local government organization.  To all of you, Thank you!

    Read more...

Opportunities for Improvement


  • We need to talk about risk, or at least I do.  I must finally confess that I have always been uncomfortable about the risk concept. I understand risk and believe it's important to think about in our audits but I also think auditors may miss important findings if they rely too heavily on a risk-based approach. 

    Read more...

Past ALGA Quarterlies

From the ALGA President

AudiTechie Column

  • Using SharePoint to Monitor ARRA - Summer 2010


    In the spirit of thematic issues of the LGAQ, we are looking at SharePoint technology through the lens of the American Reinvestment and Recovery Act (ARRA).  The focus on accountability and outcomes for ARRA funding is unprecedented and technology is facilitating the discussion.  SharePoint is a Microsoft product that allows you to customize an internal website to share and manage information.

    Read more...