Main Menu
Home
Site Index


Search the website
Email the Webmaster


Peer Review Peer Review
Education and Training
Advocating for Auditing
Auditing Awards


Funkhouser on Auditing Funkhouser on Auditing
Opportunities for Improvement
Quarterly Articles
Guides and Reports
Past ALGA Quarterlies
Contribute an article


Abstract archives Abstract archives
New Abstracts
Latest GAO Reports


Member Websites Member Websites
Join Our ListServ


ALGA Board ALGA Board
ALGA Committees
ALGA Member Services
Constitution, Minutes
Organization Topics
Benefits of Joining


Member Geographies Member Geographies


Using Outside Experts to Audit Complex Technical Models - December 2007 Print E-mail

Written by Larry Brubaker, King County Auditor’s Office,


One of the challenges of being a performance auditor is to quickly develop expertise on the program or service that is the subject of the current audit.  The variety of topics studied is what makes performance auditing fun, and has kept me challenged for nearly two decades.  However, as much as we pride ourselves on our ability to absorb these constant new challenges, there are times where the subject we are asked to audit is so complex or technical in nature, that it is helpful to bring in outside experts to assist with an audit, or even to conduct the audit in its entirety. The King County Auditor's Office's recent study of the King County Roads Services Division's Roads Concurrency Program is an example of when the use of outside expertise was crucial to the success of the effort.

Roads concurrency is a requirement of the Washington's Growth Management Act.  The Growth Management Act attempts to control urban sprawl by encouraging denser development in urban areas, while discouraging urban sprawl in rural areas.  The act also attempts to ensure that adequate infrastructure is in place before development can be approved.  The infrastructure must be available "concurrent" with development.  If the infrastructure needed to serve proposed new developments is not in place, the Growth Management Act requires that the proposed development be denied.  The amount of traffic congestion that will be accepted before development is denied is determined based on level of service (LOS) standards adopted by the legislative authority in each local government subject to the act.

As a growing county with a considerable amount of traffic congestion, the concurrency program plays a major role in determining where and when development can occur within unincorporated King County.  The King County Roads Service Division implements the concurrency program and uses complex traffic modeling software to assess whether traffic congestion meets the adopted LOS standards, both at the present time, and in the future, if a proposed development is approved.  The concurrency system and the traffic modeling software that it uses are very complex, and not well-understood by legislative decision-makers, or even by many in the Roads Services Division itself. When the King County Council was recently asked by the County Executive to make changes to the LOS standards, it had questions about the impacts of those changes on the amount of traffic congestion, future development patterns, and the need for and cost of additional road improvements.  When the answers to these questions provided by the executive branch weren't clear, the Council asked the Auditor's Office to conduct a study of the program to answer these questions.

Due to the complexity of the concurrency program, and the technical nature of traffic modeling systems, the Auditor's Office retained the services of a transportation planning consulting firm to conduct the study.  The consultant's job was to provide answers to the Council's questions about the impact of the changes to LOS standards they adopted. But in order to do so, the consultant needed to unscramble the concurrency "black box."  The consultants did so by replicating the model using the same software and inputs used by the Roads Services Division, and running the model through several iterations to replicate its results.  

By replicating the model, the consultant was able to answer the Council's questions, but the study ended up having impacts well beyond providing these answers.  For example, the process of unscrambling the black box resulted in the identification of several problems with the concurrency program and the underlying model, including technical errors and excessive complexity.  Additionally, unscrambling the black box uncovered a situation in which technical modeling practices and assumptions had significant policy implications that were hidden behind the veil of the black box. The complexity of the system and its underlying model allowed significant decisions about where and how development would be approved to be determined by technical modeling assumptions and techniques, rather than by overt decisions by elected officials.  For example, the consultants found that development was being denied in large portions of the rural areas of the county not due to traffic congestion in these rural areas, but due to traffic congestion in the urban areas.  Council members were not aware that the concurrency model had this outcome.  As a result of the study and the Council's response to it, the county is currently developing a new, simpler and more transparent concurrency program, which will allow policy decisions to be made by policy-makers, rather than by a black box.

Deciding whether to retain an expert to audit a subject of extreme complexity involves trade-offs.  The hourly rate of outside experts is typically considerably greater than the cost of in-house audit staff, and outside experts may not be familiar with auditing standards, the auditor's procedures and standards for report-writing and presentation. These factors must be balanced against the cost of ascending the learning curve if in-house staff is used. In this case, the trade-offs clearly leaned toward retaining outside expertise.

Using outside expertise usually requires a significant time commitment from audit staff as well.  This includes: conducting a competitive process to select the consultant; developing a contract and getting it approved; monitoring the consultant's work to ensure that it meets applicable standards; and assisting in report-writing and presentation. In this case, a substantial amount of audit staff time was spent on consultant selection and oversight. This is often time well-spent, however, as the skills brought to the table by the audit staff can complement the technical expertise brought by the consultant.

In short, making use of specialized expertise can be very helpful when studying complex or technical subjects, but does not eliminate the need to devote in-house resources to a project.



Users' Comments  
 

Average user rating

 

No comment posted

Add your comment



mXcomment 1.0.3 © 2007-2008 - visualclinic.fr
License Creative Commons - Some rights reserved
 
< Prev   Next >

Copyright © 1999-2006 Association of Local Government Auditors. All rights reserved.
ALGA, 449 Lewis Hargett Circle, Suite 290,
Lexington, KY 40503-3590
Telephone 859.276.0686 |
E-mail |