Responses to the Survey were due on July 31, 1997. The Subcommittee received the majority of the responses by that date, but also received a small
number after the due date. Nevertheless, all responses were included in the data base we established to compile the results of the survey. We received a total
of 133 responses.
The Subcommittee received the Survey responses by facsimile, electronic mail, and by regular mail. The 133 total responses were received via the
following sources:
| ABA Internet Homepage or Other Internet Sites | 27 |
| The Procurement Lawyer | 5 |
| Other Trade Publications | 6 |
| Miscellaneous Distributions | 95 |
The Subcommittee has included tables as Attachment 1 to this Report, which represent a breakout of the results of the Survey for each question. Question 1 requested those responding to identify whether they were from any of seven different categories. The possible responses included Private Law Firms, Government Attys, Corporate Attys, Judicial, Corporate Management/Contracts, Government Management/Contracts, and Other. As the table for Question 1 indicates, the majority of those who responded (105) were from the private sector, while 28 responses were from the Government sector.
In the subsequent tables, each question is restated in the table and is followed by the raw numbers of the responses for each of the questions that the seven different categories of those responding represented. For example, in response to Question 2, "Past performance ratings under FAR Part 42 typically are," one Corporate Atty, eight from Corporate Management/Contracts, seven from Government Management/Contracts, and one from the Other category, selected the response "Generally Inflated." The total number of times that each of the possible responses was selected is reflected in the far-right column under "Totals."
Any written comments by those responding also are included in the tables with the specific question to which the comments pertained. In addition, general comments from those responding are included after the table for Question 21.
Ultimately, the Subcommittee will use the results of the Survey to identify and make recommendations for potential improvements or changes in the current process for evaluating past performance. A report with the Subcommittee's recommendations is expected to be completed by February 1998.
We look forward to continued input and guidance from the joint Committees. Please let me know if you have any questions.
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other* | Total | |
| 6 | 4 | 12 | 75 | 24 | 12 | 133 |
| *Other: | Professional Technical Services |
| Proposal Consultant | |
| Architect/Engineer | |
| Corporate Consultant | |
| Contracts Consultant | |
| Business Development Manager | |
| Consultant | |
| Corporate Business Development |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other* | Total | |
| Generally inflated | 0 | 0 | 1 | 8 | 7 | 1 | 17 |
| Helpful | 2 | 2 | 1 | 32 | 14 | 6 | 57 |
| Unhelpful | 2 | 2 | 4 | 19 | 5 | 4 | 35 |
| Reflective of bell curve | 1 | 0 | 2 | 9 | 1 | 1 | 14 |
| Generally accurate predictors of future performance | 1 | 2 | 1 | 21 | 8 | 1 | 34 |
| Primarily negative | 0 | 1 | 5 | 10 | 2 | 3 | 20 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Inconsistency in reports/grading | 3 | 1 | 12 | 61 | 14 | 9 | 98 |
| Inconsistency in applying data | 5 | 2 | 11 | 50 | 11 | 5 | 83 |
| Fundamentally a bad idea | 0 | 1 | 0 | 1 | 0 | 0 | 2 |
| Gives the CO too much discretion | 2 | 1 | 5 | 16 | 4 | 5 | 32 |
| Evaluator's fear of reprisal for neg. reports | 0 | 1 | 1 | 7 | 13 | 2 | 23 |
| Inability to distinguish levels of performance | 1 | 1 | 8 | 40 | 7 | 6 | 61 |
| Inability to adequately
comment/ challenge | 3 | 0 | 8 | 46 | 7 | 5 | 68 |
| °Too subjective. |
| °Real issue is "fair and effective" government-wide implementation. |
| °There are no hurdles; the AF CPARS system is a good model. However, there could be a major problem without management attention, emphasis and review. |
| °Unfiltered/unchallenged comments appear with no ownership. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Comment | 5 | 3 | 10 | 69 | 22 | 10 | 117 |
| Seek CO review | 5 | 3 | 9 | 45 | 19 | 8 | 87 |
| Seek level higher review | 5 | 2 | 12 | 51 | 10 | 9 | 88 |
| Challenge the report under "Disputes" clause of contract | 2 | 1 | 5 | 21 | 5 | 2 | 36 |
| Challenge report at the GAO | 0 | 0 | 4 | 15 | 2 | 0 | 21 |
| Seek review at
ombudsman/ agency admin. hearing level | 4 | 2 | 6 | 42 | 4 | 8 | 65 |
| Seek review at forum external to agency only after exhausting remedies w/in agency | 2 | 1 | 7 | 33 | 6 | 3 | 51 |
| °Contractors should be able to obtain an independent hearing. |
| °Challenges other than at a level above the CO are very bad ideas which would grind the report card process to a halt. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Never | 0 | 1 | 0 | 2 | 11 | 0 | 14 |
| W/in 30 days of receipt | 5 | 2 | 5 | 55 | 9 | 9 | 84 |
| W/in 3 years | 1 | 0 | 5 | 13 | 1 | 1 | 20 |
| W/in 3 years of previously unavailable info becoming available | 3 | 1 | 3 | 8 | 1 | 0 | 16 |
| Anytime report becomes a factor in a source selection decision | 1 | 1 | 10 | 37 | 6 | 3 | 57 |
| ° Contractors should be permitted to provide new or otherwise mitigating information, such as corrective actions taken since the report was prepared. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Not considered at all | 1 | 1 | 2 | 27 | 3 | 4 | 37 |
| Considered at CO's discretion | 1 | 2 | 0 | 5 | 11 | 1 | 19 |
| Considered only if permitted to comment on the nature of the claim during the source selection process | 5 | 2 | 10 | 55 | 15 | 8 | 95 |
| °In the real world, reports are a means by which CO's punish their enemies and reward their friends. |
| °The report should not be considered at all if there is a formal appeal to the BCA or COFC. Mere contemplation of a claim or unresolved dispute should not be grounds for not considering the contract. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Report is signed | 4 | 2 | 4 | 27 | 11 | 4 | 51 |
| Of substantial completion or substantial performance of contract | 0 | 1 | 5 | 40 | 10 | 7 | 63 |
| Of final payment | 0 | 1 | 1 | 4 | 3 | 0 | 9 |
| Other | 1 | 0 | 0 | 4 | 0 | 1 | 6 |
| °Question not clear - desire the shortest time frame. |
| °Should be measured from completion of the review/comment process. |
| °Reports are not always signed. |
| °Substantial completion or substantial performance is too vague. |
| °Final payment is too late. |
| °Date of final payment or date of termination of performance, whichever is earlier. |
| °Agencies should be able to keep reports on long contracts longer if explained to the contractor and approved in writing 2 levels above the CO. |
| °The retention period should be measured from the date there is agreement the report is accurate or in dispute. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Deemed proprietary and not disclosed (except to the evaluated contractor) | 3 | 2 | 8 | 49 | 14 | 7 | 82 |
| Disclosed only w/contractor consent | 1 | 0 | 3 | 24 | 5 | 4 | 37 |
| Public records, disclosed under FOIA | 2 | 1 | 0 | 7 | 6 | 0 | 16 |
| Available on the Internet | 0 | 1 | 0 | 2 | 2 | 1 | 6 |
| Not disclosed after the comment period | 1 | 1 | 1 | 1 | 1 | 1 | 6 |
| °The reports should be disclosed only to the evaluated contractor, and not disclosed outside the government. |
| °Should be available on the Internet except for proprietary financial information. |
| °Deemed proprietary and not disclosed is the only possible choice to protect the integrity of the system. |
| °Deemed proprietary "and source selection sensitive" and not disclosed. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Identified | 3 | 1 | 8 | 57 | 11 | 9 | 87 |
| Identified upon request | 2 | 2 | 4 | 18 | 7 | 3 | 36 |
| Anonymous | 0 | 1 | 0 | 2 | 5 | 1 | 9 |
| °Authors of reports should absolutely be identified. |
| °Should be identified by title/level and not by name. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| 1 | 0 | 1 | 0 | 0 | 2 | 1 | 4 |
| 2 | 1 | 0 | 1 | 7 | 2 | 5 | 16 |
| 3 | 2 | 2 | 3 | 26 | 13 | 2 | 48 |
| 4 | 1 | 0 | 3 | 26 | 4 | 3 | 36 |
| 5 | 1 | 1 | 4 | 9 | 1 | 1 | 17 |
| ° Contractors need to know how to respond to data. |
| ° There is a lack of adequate notice to contractors. |
| ° There needs to be a way to eliminate/correct bad data. |
| ° Need government recognition that one low performing business unit need not infect rest of corporate ratings. |
| ° DoD needs to complete its current study by an IPT and act on recommendations for agencies to use consistent methodologies. Satisfaction with evaluation process varies widely by agency. |
| ° Government questionnaires have been so voluminous that customers won't complete them. |
| ° Should be objective evaluations against performance criteria. |
| ° There is not enough contractor opportunity to comment or refute. |
| ° Should be more timely. |
| ° Why not require 2 signatures. |
| ° The past performance evaluation and reporting process needs to be more structured and disciplined to avoid biases and inconsistencies in the process. |
| ° Annual updates to the reports are necessary. |
| ° A consistent format should be used across agencies with each contractor allowed to comment on a report and review these reports on request. |
| ° Past performance reports are the most effective method of evaluation for services type contracts for the operation and maintenance of telecommunication systems. |
| ° The comment process on the reports takes too much time. By the time the appeals process is exhausted, a large portion of the 3 year time period has run. |
| ° At least 5 adjectival bans should be available for selection of rating. |
| ° The government offered us a good report if a claim was withdrawn. |
| ° Inconsistent results among different agencies. |
| ° The parameters of evaluation or the factors must be well defined and communicated to all evaluators. |
| ° The current system should be eliminated. Currently, contractors can be seriously hurt while not being able to confront/refute providers of wrong or misleading information. |
| ° The USAF CPAR process is a "1." When questionnaires are used as part of source selections, some are used without proper contractor comment. |
| ° More standardization in requirements controlling past performance measurement, rating, and contractor comments is needed. |
| ° Contractors are sometimes not allowed to comment on negative reports, or are not made aware that a negative report was submitted. On negotiated procurements, the evaluating agency should be required to bring negative reports to the attention of the offeror and allow offeror's comments. |
| ° From what we have seen, one disgruntled person at a contract location can furnish a very negative report which is harmful to that company's efforts to acquire new business. We believe we perform better than average but the one disgruntled person who is still at the contract location is very negative. |
| ° Quit talking about it. Set valid criteria and execute. |
| ° Some agencies like DoE appear to be fair and reasonable versus some DoD agencies that appear to be arbitrary. |
| ° The process needs to be standardized across the services and agencies. |
| ° Approaches are disjointed and inconsistent. Data collection is focused on questionnaires since data bases don't exist. This makes the reclamation process particularly hard. |
| ° Contractors need to be permitted to comment and have access to all assessment data. |
| ° There is no criteria or standard used for baselining past performance. Additionally, contractor's past performance ratings or write-ups may not be available for contractor review or comment. The contractor may not be aware that such a report exists. |
| ° There needs to be more opportunity for the contractors to dispute findings if negative. |
| ° Uniform guidelines for presentation of data are needed. Evaluations should be no later than 30 days after the evaluation period. |
| ° The report should be a balanced overview where the whining of one small evaluator is not given the same credibility as the key government players. |
| ° Contractors must always be given the opportunity to comment on reports; and ensure that the evaluator understands the importance of being fact based. |
| ° There needs to be assurance that a "balanced" evaluation is conducted with both sufficient input, and from all appropriate sources, i.e., PCO, manager, etc. |
| ° The problems are implementation at the local agencies; interpretation of how used and how "no performance" is used/not used. |
| ° This should never have been established as a mandatory requirement, nor should it have ever been considered necessary to establish a database of contractor past performance evaluations. The consideration of past performance information had, in fact, been a common practice using informal processes that were tailored to individual source selection situations -- and that worked much better than the current system, with fewer problems. Now we collect tons of data, most of which will never be used; we apply past performance considerations even in situations where it will be of little use; we collect answers to common questions, rather than specific questions relevant to the particular source selection; and we are all struggling to deal with potential litigation that will, unfortunately, very likely bog us down for generations to come! My recommendation is to scrap the FAR requirements related to this whole subject. |
| ° It must be kept in mind that past performance reporting is based upon a single contract and reported by an agency CO who feels compelled to prepare such a report. The report is not a measure of contractor performance, but of performance on a single contract out of many contracts. Given that understanding, the process is OK. |
| ° Standard evaluation methods and forms should be used. Evaluators should have an overview of total contract performance. They sometimes have either tunnel vision or personal gripes, or both. |
| ° The process has potential but I don't believe procurement officials have the resources to keep the information current or accurate. |
| ° We were told by one CO that any comments on the ratings would cause review of the report by HQ staff in DC and lots of additional paperwork. Since we only took exception to one of the rated areas, we decided not to pursue the issue so as not to upset the CO. Another CO said they wanted to provide past performance data verbally with no paper files. |
| ° The contractor also should evaluate the contract administrator and others involved. The contractor also should evaluate himself as to performance, timeliness, and contract interpretation. |
| ° Contractors should be notified of discrepancies immediately to allow correction of the problems -- not just at the time of the report. |
| ° I think more weight on "controlled" past performance is appropriate, but with contractor right to challenge. |
| ° The process is too cumbersome to obtain contract mods for Government schedule errors -- contracts are accepted and performance is impacted. |
| ° Source selection officials should solicit contractor comments on all past performance reports considered as part of an evaluation for contract award. |
| ° On DO/IT services contract individual orders (or groups combined at CO direction) should be evaluated. |
| ° Needs consistency in use. Question 10 Comments - Cont'd
|
| ° COs and contractors should comment and it should be available to everyone on the Internet by contractor and principal name. |
| ° Agencies requesting past performance information should limit the amount of information requested. Currently, evaluation requests have included information which is superfluous and not specifically tied to past performance. All wastes the evaluator's time. |
| ° Consistent guidelines are needed on measuring past performance. Also, need to see the ruling when reports are disputed. |
| ° Various factors under the report are somewhat unreliable, but it is better than not having them to consider at all. |
| ° Specifics need to be given for any rating below satisfactory. |
| ° Objective detailed past performance reports must be available to COs to allow them to determine past performance. Currently, these reports and the process is much too subjective and "personal." Standard evaluation formats must be developed. May I suggest the Air Force's CPAR? |
| ° The factors should be standardized to make it less subjective. |
| ° The reports are too subjective with no ability to really differentiate. Past performance should be just a minor risk factor for evaluation purposes. |
| ° The reports are far too subjective. Make evaluations specific. Follow FAR Part 42. |
| ° Verifiable metrics would result in more objective decisions/ratings. |
| ° Technical should be separated from contract admin evaluation. Technical is not given specific weight, especially on start-ups. Contractors should not be penalized in reviews for taking admin risks when technical performance is high. CO needs to be balanced by the technical director. |
| ° Need to improve consistency, allow contractors to read and refute, and compel COs to treat reports as more than pass/fail. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | ||
| Yes | 3 | 0 | 6 | 39 | 3 | 6 | 55 | |
| No | 1 | 1 | 4 | 18 | 3 | 4 | 31 | |
| # of days | * |
| °There are three types of reports (i) formal contract performance evaluations, i.e., CPARS (we get to comment on these); (ii) questionnaires sent to individuals on "relevant" contracts by the source selection authority (we generally don't get to comment; and (iii) data contained in government and industry data bases, i.e., MOCAS, GIDEP, Navy Reliability Center (we generally don't get to comment). |
| °Who fills out the report is important. Each person has a unique perspective. In a systems acquisition, only the Program Manager has the full view. |
| °Not given the opportunity to comment. |
| °We have never been given the opportunity to comment. |
| Private Law Firms | Gov't Atty | C o rp.Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Contracting officer | 3 | 0 | 8 | 41 | 4 | 7 | 61 |
| One level above CO | 2 | 1 | 2 | 17 | 1 | 6 | 27 |
| More than one level above CO | 1 | 1 | 1 | 8 | 0 | 2 | 13 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Written | 4 | 1 | 8 | 53 | 3 | 10 | 77 |
| Affidavits | 1 | 0 | 0 | 2 | 0 | 1 | 4 |
| Ltrs./materials from third parties | 1 | 0 | 1 | 8 | 0 | 2 | 11 |
| Oral presentations | 1 | 0 | 3 | 10 | 0 | 2 | 15 |
| Oral discussions /meetings | 3 | 1 | 5 | 28 | 0 | 3 | 39 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Nothing | 2 | 1 | 4 | 23 | 1 | 2 | 32 |
| Agency responded orally | 2 | 0 | 1 | 11 | 0 | 4 | 17 |
| Agency responded in writing | 2 | 0 | 3 | 13 | 2 | 6 | 25 |
| Revisions were made to original report | 1 | 0 | 2 | 11 | 0 | 4 | 17 |
| Contractor's comments were appended to original report | 2 | 0 | 2 | 14 | 1 | 4 | 21 |
| Don't know | 1 | 0 | 6 | 19 | 1 | 2 | 28 |
| ° Generally no information is given to contractors. Contractors hear or receive comments 3rd hand. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| 1 | 0 | 0 | 0 | 3 | 0 | 2 | 5 |
| 2 | 1 | 0 | 1 | 6 | 1 | 4 | 12 |
| 3 | 1 | 1 | 1 | 15 | 0 | 2 | 20 |
| 4 | 1 | 0 | 4 | 13 | 0 | 0 | 18 |
| 5 | 0 | 0 | 2 | 8 | 1 | 1 | 12 |
Number | Forum | Successful [Y/N] |
90 | CO/Agency | Yes = 32 No = 52 |
| 0 | GAO | |
| 0 | Court | |
| 0 | Administrative Board | |
| 0 | Other |
| °A number of challenges have been made within various DoD components. |
| ° There have been mixed results at the agency level.
|
| ° Comments were appended to the report prior to placement in the AF data base.
|
| ° We have had mixed results. |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Increased workload | 0 | 2 | 0 | 2 | 7 | 1 | 11 |
| Made source selection more burdensome | 0 | 1 | 1 | 2 | 3 | 1 | 7 |
| Failed to alter contractor performance | 0 | 1 | 1 | 1 | 1 | 0 | 4 |
| Enhanced my ability to select superior contractors | 0 | 2 | 0 | 0 | 8 | 1 | 11 |
| Helped to eliminate poor performing contractors | 0 | 2 | 0 | 1 | 9 | 0 | 12 |
| Produced improvements in contractor performance, attitude etc. | 0 | 2 | 0 | 0 | 6 | 2 | 9 |
| Not been worth the effort, resources, etc. | 0 | 1 | 0 | 3 | 4 | 0 | 8 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Technical reps. | 0 | 3 | 1 | 3 | 14 | 3 | 23 |
| End users | 0 | 1 | 1 | 3 | 12 | 1 | 17 |
| Plant reps. | 0 | 1 | 1 | 0 | 2 | 1 | 5 |
| Auditors | 0 | 0 | 0 | 0 | 3 | 1 | 3 |
| Counsel | 0 | 0 | 0 | 0 | 2 | 1 | 2 |
| Contracting officers and specialists | 0 | 2 | 0 | 1 | 15 | 3 | 20 |
| Non-government services | 0 | 0 | 0 | 1 | 2 | 0 | 3 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Yes | 0 | 2 | 0 | 3 | 4 | 1 | 10 |
| No | 0 | 1 | 1 | 2 | 12 | 2 | 17 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Yes | 0 | 1 | 0 | 1 | 3 | 0 | 5 |
| No | 0 | 2 | 1 | 4 | 12 | 2 | 20 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Yes | 0 | 0 | 0 | 0 | 3 | 1 | 4 |
| No | 0 | 0 | 0 | 2 | 1 | 0 | 3 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Yes | 0 | 3 | 0 | 1 | 8 | 1 | 12 |
| No | 0 | 0 | 1 | 3 | 6 | 2 | 12 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Yes | 0 | 0 | 0 | 2 | 3 | 0 | 5 |
| No | 0 | 0 | 1 | 1 | 5 | 0 | 7 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Yes | 0 | 3 | 0 | 1 | 5 | 0 | 9 |
| No | 0 | 0 | 1 | 3 | 7 | 2 | 13 |
| Private Law Firms | Gov't Atty | Corp. Atty | Corporate Mgmt./Contracts | Gov't Mgmt./Contracts | Other | Total | |
| Yes | 0 | 2 | 0 | 1 | 6 | 1 | 9 |
| No | 0 | 1 | 1 | 3 | 9 | 0 | 14 |
| Return to Federal Claims and Remedies Committee | Return to Judical Remedies Committee |