Association of Local Government Auditors

Professional Auditors write and issue audit reports to improve decision-making and enhance public accountability.  As one of more than 300 ALGA member organizations, Portland has entered the Knighton Awards program many times and fared pretty well.  In this article, I'd like to make some suggestions for Knighton submissions based on our experience, and information I gathered from Knighton judges and committee members.

But first, a disclaimer: The author may, or may not, know what he is talking about!  And with that, read at your own peril.

I offer that disclaimer in light of the following incident that happened in early 2005.  The place:  City of Portland Audit Services Division library.  Our office met about submitting a report for the awards program, and the group was optimistic about the chances of our Knighton submission to win.  The report was one I had been in charge of.  My response was that there was no way a report of Police Detective effectiveness would win anything; it had no financial findings and everyone knows that it takes big financial findings to win a Knighton award.

The report won.  Same for another report I in-charged in 2007, about which I felt only marginally better upon submission. 

It did get me thinking, though, about what judges actually look for when they are rating these reports.  So when the ALGA Awards Program Committee asked for a volunteer to write an article, I thought this would be a good chance to answer some nagging questions.

My working hypothesis was that an award-winning audit likely displays the same characteristics as any well-written audit we all strive to produce. 

Let's see.

So first, like any good auditor I pulled together the Knighton criteria and began interviewing those with more knowledge of the process than I:  Awards Program Committee members and past judges.  I used some of their comments in this article.  But let's be clear, the interpretation of this information is mine and any error or misstatement is totally on me.

So what are the judges told to look for?

First off, the judges look for the qualifying material outlined in the Awards Announcement.  Judges first make sure the report contains a standards compliance statement.  Then there must be a "consent to publish" statement in the cover letter.  If these two items aren't there, the submission is automatically disqualified.

In reviewing the reports, judges are told to equally weight the following attributes (also found in the annual announcement) on a twenty point scale:

• Scope and potential for significant impact on the auditee's operations
• Persuasiveness of conclusions
• Focus on recommendations for effective and efficient government
• Clarity and conciseness of communication style, and
• Innovation in presentation

Beyond that, the judges use their own perspectives to rate the reports.  The equal rating scale tends to balance out personal preferences to a degree.  Also, the judges meet via conference calls to discuss their ratings.  After talking with several judges and reviewing comments, I've concluded that, whether they mean to or not, many judges have a hierarchy of criteria in mind when they review reports.  Some of the criteria seem to be more important than others, even though in the end the final ranking of the report and whether it wins or not may depend on other factors such as the number of reports submitted in the category and their relative quality (in other words, the strength of the competition).

Clarity and conciseness count

This appears to be the number one issue on most judges' minds.  It makes sense because a clear, concise, interesting report is easier for a judge to read than a long, hard to follow, overly detailed and repetitive report that blathers on and on and on.  It just so happens that is likely also the case for your average Mayor, Audit Committee member, newspaper reporter and citizen.

What "clear and concise" means, however, is left up to the individual judge.  Here are some of the comments mentioned most often:

• A clear message is critical to a well judged report. The message should be clear from the beginning and all points, graphics, conclusions and recommendations should tie to that message. The actual message should be short, like the tag line to a movie. In Portland, we title each report with the message and work to emphasize it throughout the report.
• Shorter is generally better. One thing that seems to bother judges is a long report; in particular, one that seems to be longer than necessary. As more than one person told me, people don't have time to read 25 page reports, much less 90 page reports. A report that is too long detracts from its impact. I think I can say from personal experience of over twenty years writing audit reports, shorter reports tend to get more attention and action than long tomes. And remember, less technical detail and more message.
• Reports should have a logical, natural flow. Again, a flow that supports the message. I usually think and write in a deductive pattern. That is, I present general conclusions, and then I present detailed material that supports the general statements. Others think and write more in an inductive way, presenting details first that lead up to a general conclusion. Whatever way you choose, make sure you stick to it. Vacillating between the two styles is the surest way to bog down a reader.
• Easy to read is good. Simple language and supporting graphics, with good organization is a winning combination. Writing a report with words you haven't used since grad school may make you feel better, but it won't help you get your point across or keep your readers' interest. Judges also commented on reports' use of white space to complement a report's organization and formatting, making reports easier to read.
• "Old school" report presentation is out. Sorry for those of you who think you have to list the five elements for every finding. The five elements should be there, but need not be overtly listed. A report that consists of findings and elements can be pretty dry and boring. Some judges might like this, but overall, this is not the best approach. Try to tell a story that incorporates the elements, and as always-the message.
• Too many charts could be overkill and detract from the report. Graphics are good when they support a main point, but they should be simple and easy to read. With charts, we try to make each one with no more than one line or bar series, and never more than one axis. I remember many years ago when we talked about modeling ours after clean charts we saw in the Economist magazine. That seems to work for us. You might look for a source with a simple model you like, then be consistent with the format. Different formats will make the reader work too hard.

Scope (and impact) is very important

Scope and impact seem to go hand in hand in the minds of the judges I talked with.  The best audits had the most potential for significant impact.  This was not just a function of dollar findings; most of the audits I reviewed last year didn't have explicit dollar findings.  In addition to dollars and cents, judges think about the potential to positively affect public health and safety.  Audits on topics that have rarely, if ever, been covered also get attention.

Recommendations

Recommendations need to be clearly tied to the findings in the report.  They also need to be ready to be implemented.  One judge said he puts weight on the acceptance of the recommendations by the decision-makers.  Another judge said she read multiple reports where recommendations did not tie to findings or objectives.

What is innovation in presentation?

The one subject no one has a clear grasp of is innovation.  What does this mean?  Some commented on the use of graphics, white space, fonts, and report organization. Some think it refers to innovative audit techniques.  Others think it means a topic that hasn't been audited.  In any case, blazing new trails in auditing can help a report's chances in the awards program.  A new graphic, a new angle or approach to studying a problem, or a compelling case study can all help convey an audit's message and help the presentation of audit results.

What should the Knighton Award submission include?

The submission process has been simplified and streamlined.  Each Knighton Award submission must include a completed 2011 Knighton Award Submission Form.  An electronic version of the form is available on ALGA's website (www.governmentauditors.org).  The form contains five sections:

• 1. Section A captures general and contact information about the submitting audit shop and the audit shop size.
• 2. Section B affirms that the audit organization has met the minimum eligibility requirements, and gives ALGA permission to share award winning reports with its members by placing them on its website.
• 3. Section C seeks additional information on how your report submission meets or exceeds three of the five Knighton Award criteria. Provide a short description or background information for the judges that address how your:
  • a. Audit scope has the potential for significant impact, and is responsive to the needs and concerns of decision-makers and/or the public;
  • b. Audit recommendations are feasible, and will make government programs more effective and efficient; and
  • c. Audit scope, methodology, recommendations, or report format is innovative.
  • d. Note: This section replaces the Transmittal Letter required in previous years. In addition, supplemental information (such as newspaper clippings, news reports, etc.) submitted in conjunction with the report to illustrate audit impact, is no longer allowed.
• 4. Section D, which is optional, asks for your participation in the awards program transparency efforts. Your willingness to participate (or not participate) will have no impact on your submission or the judging process.
  • a. ALGA members have indicated they are interested in reading the reports submitted for an award. Please indicate if you give ALGA permission to share your report submission on its website if you do not win an award.
  • b. The Awards Program Committee is trying to understand the amount of effort expended on the audits submitted. Please share the number of hours spent on your submitted audit.
• 5. Section E captures the Chief Audit Executive's approval of the Knighton Award Submission Form and affirms the accuracy of the information provided.