By Karen L. Garst
Many bar associations face the following dilemmas: a deficit that demands cutbacks in programs or services and a desire to expand current programs or develop new ones.How are decisions made in these instances? All too often they are based on a program’s perceived value. Perceptions about a program might stem from opinions of board members or staff, or from anecdotal information shared in the hors d’oeuvres line at the last social function. We all can be influenced by a particularly strong story that we hear from someone we respect. Such influences often drive our decisions. But do we ever stop to ask if the perception is valid? Will other people agree? Is there any objective evidence to support the perception?
In the past, the Oregon State Bar Board of Governors often made decisions based on similar perceptions. Board members or staff presented ideas for programs based on their own personal experiences or discussions with other bar leaders. If an idea "seemed good" or if a board member was particularly forceful, a program was born. While the bar has developed many good programs, some new programs were created with little objective information or knowledge about the value they would add to the bar’s mission. When it became necessary to scale back programs, this data was missing. To complicate matters, a previous strategic planning effort left the board with a long to-do list, but no method for making priority decisions.
When I came to the Oregon State Bar in 1996, President Dennis Karnopp and outgoing Executive Director Celene Greene had already started to think about a program measurement process. In my previous position, I worked with the Oregon Benchmarks project, which is a statewide effort to establish goals and objectives in all service categories within state government departments--in areas ranging from clean air to teenage pregnancy. Each department of the executive branch must define specific measurable outcomes for the programs and services it provides. A state board called the Progress Board oversees the benchmark process.
Since I worked in the area of post-secondary education, I was familiar with the benchmarks process, which used goal statements and measurable outcomes, and I was able to draw from that work to establish our process at the Oregon bar.
In preparing for my first board retreat, I hired a consultant who introduced the board to the basics of program evaluation. We needed to ask ourselves the following questions: What is an outcome? How do you focus on results? How do you begin the measurement process? During the next several months, board member Mike Phillips led the board’s Long Range Planning Committee as it developed an "evaluation template." The template was the outline for the information we needed in order to assess our programs. It includes a brief description of the program and its goal statement, which is similar to a mission statement. We asked: What is the primary focus of the program? What does success look like? What is the key result that is expected? Next, we included some basic information about the program: Number of staff. Revenue. Expenses. Then came four or five key outcomes with a statement about how they could be measured, as well as the target outcome.
The process of developing the template took several committee meetings of working with a few programs to test it out. Since our bar developed the template, I’ve learned that there are many associations and public sector organizations working on similar program measurement processes. We relied on the research provided by the consultant we hired, but have used additional research to revise the template.
Goal statements
The first step in the process of using the evaluation template was to create goal statements for each bar program. Our staff developed goal statements for support services such as accounting and human resources, while the board drafted goals for substantive programs.
For example, the board adopted the following goal statement for our CLE Seminars program: Enhance the knowledge and skills of Oregon attorneys through production of seminars and other educational offerings made widely available on a broad range of legal topics, at different skill levels, in a fiscally responsible manner. Assure a competent bar through the provision of quality programming.
To develop this goal statement, we discussed the specific role of our CLE seminars in comparison to those offered by sections and other providers. We also took a look at the state’s MCLE requirements. We concluded that our niche is not one of specialization since section CLE programs fill that market. Rather, our programs needed to cover a broad range of topics. As a mandatory, statewide organization, it is important to address members at different skill levels and offer widely available programs.
The goal statement adopted for Accounting, a support service, was: Continually improve service to customers, internal and external, in the most efficient and accurate manner possible.
These goal statements are similar to organizational mission statements. Ideally, goal statements are developed with input from program managers, staff and bar committees. A side benefit to our process is the opportunity for program managers to discuss the fundamental purpose of their programs with board members.
Outcomes
While several managers applauded this opportunity to discuss their program with the board, it was clear that more needed to happen. If we had simply stopped with the goal statements, we would have failed to provide adequate enough clarity and direction to program managers. Our next challenge in developing the evaluation template came as we began to outline specific outcomes in line with the goal statements.
Continuing with the CLE Seminars example, one specific outcome is: "Ensure that the CLE Seminar department is administered in a fiscally responsible manner." For Accounting, we decided one outcome would be to "decrease the time for processing orders." A specific outcome for our Client Security Fund is to "increase the amount of money recovered from lawyers for whom CSF claims have been paid." Action verbs (maintain, increase, decrease, ensure) can assist you in writing outcome statements. Using these verbs allows you to focus on your end result rather than the tools you will use to achieve it.
Measuring performance
After the board adopted specific outcomes, I worked with the program managers to decide just how we would measure program performance. This was accomplished by determining whether we had achieved each of the outcomes.
There is an adage in business today that if you can’t measure it, you can’t improve or change it. This is true even when talking about good results. Imagine a chef who stirs up a great recipe but can’t repeat it because she didn’t keep track of the ingredients. While less tangible, a bar’s programs and services must be held accountable through verifiable measurement of achieved objectives. That doesn’t mean that managers turn into bean counters, however. There are endless ways to assess results--surveys, direct observations and, yes, even counting beans.
As an example, for the CLE Seminars outcome, we chose a break-even budget as an appropriate measure of fiscal responsibility. In the first two years of the program measurement process, CLE Seminars finished 5 percent over budget one year and 2 percent under budget the other year. Evaluating these results--an essential element of the program measurement process--caused us to redefine what "break-even" means. We created a small reserve cushion within the budget to help even out the peaks and valleys, small as they were. Also, as a result of the evaluation process, we decided to create an additional budget reserve for research and development. This will allow us to better prepare for future developments in CLE program delivery.
In order to meet our outcome for Accounting, we took a thorough look at the order processing system. The target was to "reduce processing time by moving to online entry of phone orders, re-engineer the paperwork flow to minimize steps in the process, and to keep return rate to less than 1 percent of orders."
When selecting a measure, it is not necessary to look at everything in a program that can be measured. Instead, an "indicator measure" will serve the purpose. Just as Oregon’s spotted owl was an indicator of the health of an old growth forest, the return rate in our accounting department would be the indicator measure to show us if the new process was an improvement. The year-end evaluation of the outcome indicated that we "processed 10,489 orders in the 11 months ending November 1997 with error rate of less than 1 percent (.05 percent)." Please note that outcome statements are not statements of strategies, action plans or activities. They should be results oriented.
As a final example, let’s look at one specific outcome for the bar’s Client Security Fund. In this case, the measurement was fairly simple. We compared recoveries at the end of the year to the recoveries of the previous year. The comparison showed a recovery of $6,000+ more than the previous year.
While our program performance measurement is in its infancy, the template has already been used to change processes within programs, as shown by the previous examples. All but one of the bar’s programs have developed outcomes that are measurable. The accounting department’s new ordering process reflects such changes. In addition, we have used this method of measurement to modify programs. For example, after a membership survey of bar programs, we decided to change our annual meeting to a biennial convention. We also used the evaluation template to create a new communications program and a new section for solo and small firm practitioners. In the case of the solo and small firm section, I met with a group of interested solo and small firm practitioners and, together, we identified the key outcomes that they wanted the section to achieve. Those outcome statements went to the bar’s Board of Governors as part of the agenda item to create the new section. For now, the template process has not resulted in the elimination of a bar program.
We now link program outcomes to the performance appraisal of each manager. Up to 60 percent of a manager’s evaluation depends on achieving the key outcomes for his or her program. Each year, the board receives a complete evaluation of each program. A board committee has placed each program on a two- or three-year cycle for a more comprehensive review by the board.
We have a lot to learn about the program measurement process. However, our initial steps have given us the concrete information we need to assist the board in making decisions based on objectives, rather than perceptions. The author is the executive director of the Oregon State Bar. Before joining the bar in 1996, she served as executive director of the Oregon Community College Association for eight years. During her 16 years in membership service and government relations, she worked as a field representative for the American Federation of Teachers in Oregon. Garst holds a doctorate degree in educational curriculum and instruction and a master’s degree in French.
For additional information about the program evaluation process or a copy of the evaluation template, Karen Garst can be reached at the Oregon State Bar, 5200 SW Meadows Ave., Lake Oswego, OR 97035; 503/620-0222, extension 312; or kgarst@osbar.org.
