1. Introduction

1.1 Why client feedback is important?

Feedback from clients regarding your seminar or workshop can be informal or formal, unstructured or structured. It can range from a simple question such as "How useful did you find the training today?" to a multi-page questionnaire.

Collecting information from your clients through a well-designed process does have a number of advantages. These include:

1. Making improvements and enhancements to your training to allow you to better meet the learning needs of your clients and thus ensure customer satisfaction.

2. Identifying and responding to changes in clients' needs and work environment. This is particularly true if you have a seminar that you have been giving over a long time.

3. Allowing you to adjust your training to specific market segments. Feedback can help you identify subtle differences in needs, e.g., time management for supervisors, time management for technical professionals, time management for entrepreneurs.

4. Identifying opportunities for new and/or enhanced training products and services. For example, after a general Internet course, people may want a course on design Web pages.

5. Being required by sponsors or governments. Many of our public seminars have been sponsored in part by government departments. They required a detailed evaluation report at the completion of the projects.

6. Assisting in the measurement and improvement of the effectiveness of marketing and promotion programs.

7. A good evaluation program can assist to differentiate you from the competition.

8. Positive evaluation results can be used as a powerful marketing tool.

1.2 Some disadvantages and problems with client feedback

More formal, structured feedback systems do have a downside. These include:

1. It takes considerable expertise, time, effort and money to design, analyze and use formal feedback systems. You have to do a cost-benefit analysis for your operations to determine the investment you are willing to make in getting client feedback.

2. If additional costs are incurred in obtaining feedback, these may have to be passed on to your clients. This may make your services less cost competitive.

3. Feedback needs to be done right - otherwise the results are useless, or even more dangerous, misleading.

4. Clients get tired of filling out questionnaires. Any survey after the seminar or workshop often results in a lower response rate.

5. Less than perfect results are hard on the trainer's ego.

6. May not provide all the answers, but will reduce the risk.

1.3 When to collect client feedback?

Client feedback can be obtained at a number of points during the training/learning process. These are:

1. Before the training. Data is collected from a variety of sources about the content and preferred format for learning.

2. During the training. Good trainers constantly monitor the progress of their training and their clients, and make necessary adjustments.

3. Right after the training. This is the most common approach where the clients are asked to fill out an evaluation form.

4. Way after the training. Some time is elapsed before the clients are contacted to determine how effective the training was.

An ideal client feedback system is one that obtains input from all the above stages of the learning process.

2. Strategies and Techniques to Obtain Client Input

Let's take a look at some of the data gathering strategies and techniques that can be used to obtain client input at various stages of the training/learning process.

2.1 Before the training

1. Interviews. If the training is an in-house workshop, then interviews can be conducted with the supervisors and a sample of the incumbents to determine what the learning needs are. If the seminar is a public one, interviews with a sample of potential clients (obtained from your customer database) can help design an appropriate session.

2. Task analysis. This can be done either through interviews or surveys. A list of tasks is compiled. The job incumbents and their supervisors then rate the important and frequency of each task. This provides a list of the critical knowledge, skills and competencies that are required to be effective for that job. This serves as the basis for designing the training and learning experiences.

3. Needs assessment. We have sent out a general questionnaire to a prospective list of clients asking them whether they would be interested in attending a training technology conference. The questionnaire asked about preferred location, times, length, and topics. Needs assessments can be simple or comprehensive but are a good way of getting marketing research data.

4. Focus groups. This involves getting together a representative sample of the clients your wish to serve, and asking them questions regarding your proposed seminar or workshop. The process needs to be structured and feedback recorded.

5. On-line focus groups. Using the Internet's chat capability, focus groups can be conducted on-line. The advantages are that you can get people together from anywhere in the world, there are no travel or room costs, and the discussion is recorded. The disadvantages are that not everyone has access or is comfortable with using the Internet (e.g., ability to type), and that body language is absent.

6. Pilot projects. One way to get client input into a new seminar or course is to run several pilots first. It is important to schedule several in order to make changes and adjustments based on client feedback and experience with the session. Tell people that this is a pilot program, and that from time to time you may stop the session to get their comments. The pilot workshops and seminars are also a good way to test the potential market demand.

7. Competition analysis. Attend a seminar or workshop offered by a competitor. Count the number of people attending, talk to the participants and take good notes on how you can make your workshop better than your competitor's.

8. Other tools. These include advisory boards, job descriptions, performance evaluations, competency profiles and critical incident analysis.

2.2 During the training

Here are some ways to get feedback during the training session:

1. Ask. Good trainers are always asking the participants questions to assess how the session is going.

2. Body language. Monitor the body language of your participants to determine when they need a break, or an activity, etc. However, be cautious about misreading body language, especially in evening workshops or when working with groups from different cultures.

3. Instruments. In our time management workshops, early in the morning the participants do a time mastery profile. The results of their effectiveness in the various aspects of time management are tallied on a flip chart. The rest of the workshop is based on areas of greatest need. The workbook and materials we have available cover all the possible topics, but with the time available we only cover the priorities for that group of participants.

4. Professional evaluation. In one of my workshops, I had an Associate, who happens to be a superintendent of schools sit in. I asked him to give me an evaluation at the end of the session. Since he was a professional at evaluating teaching performance, his feedback was extremely useful. If a school superintendent is not available, school principals, members of Toastmasters and other experienced trainers can be used as expert evaluators.

2.3 After the training

There are a number of strategies that can be used to collect feedback from clients both immediately at the end of the training session and afterwards. These are:

1. Questionnaire or survey. This is probably the most common approach used to obtain feedback. Participants are asked to fill out a form providing feedback on the various aspects of the training. Questionnaires and forms can also be mailed out after some time has elapsed. These are useful in obtaining feedback on whether the knowledge and skills learned at the training are being applied on the job. Questionnaires can also be sent to the participants' supervisors or managers asking them whether their performance has improved.

2. On-line forms. With the availability of the Internet, an evaluation form can be put on a Web site with requests for people to answer. On-line forms are also a good way to gather general market intelligence regarding training needs.

3. E-mail. An evaluation form or questionnaire is simply e-mailed to participants who are asked to complete it and return it. On issue with e-mail forms is confidentiality - you know who sent you the e-mail.

4. Telephone calls. Another option for collection of feedback from clients is to call a sample of them after the training. The questions can be either informal, or better still, have a list of questions that you ask them. The telephone calls can also be used to interview the supervisors and managers.

2.4 Continuous improvement process

If you are offering the same course or training repeatedly over an extended period of time, you can set up a continuous improvement process. The process as shown below involves doing a needs assessment prior to the training, an evaluation immediately after and another one some time after the training. The evaluation results are used to constantly monitor, improve and enhance the training.

3. Asking the Right Questions

3.1 General considerations

Here are some general things to keep in mind when asking questions:

1. Collect only data that you can use. Some people ask questions from their clients that have no relevance to improving the training. Know in advance how you will use each unit of data in making decisions, planning, enhancing content or altering delivery.

2. Collect enough data to make a decision. If you are asking people whether they would like to have workshops in the evenings, be sure to find out what days, what start times, how long, etc.

3. Try and keep your questions consistent across courses and training activities. Develop effective questions and stick with them if possible. This enables you to collect comparative data to identify trends and compare performance over time and among different seminars and target populations.

4. Take advantage during the collection of feedback to collect market research data. Find out what other courses they would be willing to take, preferred times, etc. Adding a few questions to an evaluation survey is a very efficient way to collect valuable market intelligence.

5. Either use the same evaluation form or some means of linking responses from respondents if multiple forms are used. This allows greater opportunities for cross-tabulation analyses.

6. Respect respondents' confidentiality, especially with internal training activities. Either have the questionnaire answered anonymously or only present group results.

7. A combination of structured and open-ended questions seems to provide the best results.

3.2 Types of information to collect

Here are some types of feedback information that can be useful to collect:

1. Levels of achievement and performance. These typically include the following:

• client satisfaction with the training experience
• perception or objective measure of the learning that took place
• some measure or perception of the change in performance as a result of the training
• overall impact on the performance and profitability of the unit.

Satisfaction is the easiest to measure and as a result it is most commonly used. In many situations this measure is enough. If you are offering a public seminar and your clients are satisfied, then no additional measures may be required. However, if you are an internal trainer, the organization may demand evidence of a "return on investment" of the training experience.

2. Content and topics covered. Asking about specific units and topics covered during the training will enable you to add, delete or modify the content.

3. Format and delivery. This provides feedback on changes needed to the format, delivery and activities in the seminars.

4. Demographics. This allows you to determine whether the training needs of various groups differ. This allows the fine-tuning of your training to each group.

5. Previous knowledge and training. This information is useful in determining design and level of your training. For example, an Internet workshop will be very different depending on the degree of computer experience the participants have.

6. Intentions. This information is of interest to organizations or governments that are using training to increase awareness and/or bring about change. The risk is that the participants may not follow through with their intentions.

7. Market research questions. These questions include topics, preferred delivery and costing data.

3.3 Examples of useful data elements

Here are some sample questions to collect the data described above.

1. Content

How effective were each of the following seminar components? (List the different components and have them check on a four or five point scale).

How well were the following seminar objectives achieved?

What was the most significant idea you learned from this seminar? (Open-ended question).

2. Delivery and format

How effective were the following delivery aspects of the seminar?

Ask about the following:

• Preparation of the instructor
• Organization of the seminar or workshop
• Hand-out materials and notes
• Amount of material covered
• Enthusiasm of the instructor
• Appropriate use of audio-visual resources
• Group exercises and activities
• Answering participants' questions
• Pace of the seminar
• Time allocated to the seminar
• Ability of the instructor to communicate
• Knowledge of subject by the instructor
• Appropriateness of physical setting and facilities
• Overall instructor performance

Check the area(s) in which the seminar could have been improved in order to better meet its objectives and provide you with the knowledge and skills you were looking for. (Provide a list of options).

In your opinion, what part(s) of this seminar was superior? (Ask as an open-ended question).

3. Satisfaction

Overall, how satisfied were you with this seminar?

Based on your experience with this seminar, would you take another (company name) seminar?

Based on your experiences with this seminar, would you recommend a (company name) seminar to a friend or co-worker?

If you have a satisfaction guaranteed policy, another measure is to keep track of how many people asked for their money back.

4. Promotion and marketing

How did you find out about this seminar?

Provide a checklist that should include:

• Program guide
• Brochure in the mail
• Brochure that was picked up
• Company or other organization offices
• Friend or neighbor
• Newspaper ad or article
• Radio or television
• Internet (e-mail)
• Internet (Web site)
• Any other promotion that you might have used.

How could the promotion of this seminar be improved? (Ask as an open-ended question).

5. Intentions and impact

Based on your experience in this seminar, how likely are you to get the Internet in the next six months?

How likely are you to set up a World Wide Web for your business in the next 12 months?

Based on what you learned at the workshop, which of the following actions are you likely to undertake in the next six months? (Provide a list of desired achievements)

6. Previous knowledge and experience

What is your level of knowledge and experience with computer technology?

How many computers do you have?

How often do you use on-line services such as the Internet?

How many computer-related seminars have you taken before this one?

Do you or your company have an Internet Web site?

7. Demographics

What is your gender?

What is your age category? (Try and use categories similar to the ones used by your Census Bureau; it permits better comparisons with the general population).

How many years have you been involved in managing a business?

What are the main sources of income from your farming/ranching operations?

What level of formal education have you completed?

Which of the following categories best describes your gross farm income for last year?

Which of the following best describes your role and position in respect to answering this questionnaire?

8. Market research

How interested are you in attending similar conference in the future?

What is the best time of the year for you to attend these types of workshops and conferences?

What other seminars and workshops would you be interested in taking? (Check all that apply)

What length of workshop or seminar is best for you? What is the best time of day for you to attend a workshop?

What is the best time of the week for you to attend a workshop?

What is the best month(s) for you to take workshops or seminars?

What is the most you would be willing to pay for a half-day workshop?

What is the most you would be willing to pay for a one-day workshop?

How important to you is it to have a family discount for workshops and seminars?

8. Research information

This can include the collection of information that can be used for research purposes. For example, if you are doing a public seminar on time management, collecting and correlating their time management scores can provide some interesting insights into problems and issues related to gender, occupational groups, etc.

3.4 Some design considerations

Here are some design considerations that apply mostly to evaluation forms and questionnaires:

1. Desk-top publish your questionnaire. It makes it look more professional, but more importantly, you can ask a lot more questions because the questionnaire looks shorter.

2. Use close-ended questions. Well-designed closed-ended questions with many alternatives will give you more data, more consistent results and make it much easier to analyze. Data from categories also give you many more options for statistical analyses.

3. Include open-ended questions. Always add in a few opened-ended questions and encourage people to elaborate or expand on their responses to structured questions. Make this clear in the instructions.

4. Pre-test your instruments. It is always advisable to pre-test your survey or questionnaire to see that the instructions and wording are clear and understandable. Keep in mind the literacy levels of your participants and what you are asking of them.

5. Pre-code your questionnaire. This makes the data input much faster and requires less instructions and supervision to get the data into an electronic file.

6. Always put the demographic questions at the end. Begin with the evaluation of the content first, followed by an assessment of the format and delivery. Ask market research questions and demographics last.

7. Use a booklet. You can include a lot of questions on a four-page booklet (11 by 17 folded over once). It looks professional and yet not too long.

8. Provide contact information. If you are doing a needs assessment or a questionnaire that has been mailed out, always include a name and phone number of someone they can call if they have a question.

9. Use a cover letter. This applies for questionnaires and surveys done at other than immediately after the training. The letter should include the reasons for the survey, what will be done with it, instructions as to when and where to return it, and a contact name and phone number. The author has found that providing a toll-free fax number is a good way to encourage participants to send back their surveys.

10. Offer prizes. Offering a book or other relevant prize encourages people to complete the evaluation form.

4. Analyzing and Interpreting the Results

Here are some suggestions and considerations for analyzing client feedback data.

1. Use statistical software. Use a proper statistical program such as SPSS (Statistical Package for Social Sciences) rather than just a spreadsheet. SPSS allows a much broader range of statistical analyses, enabling you to get the most information from the data.

2. Check your data. Once the data has been entered into an electronic file for statistical analyses, perform some basic checks to ensure it is correct. One way this can be done is by checking for "out of range" data. Correct the data before your proceed. Also make sure that your labels refer to the right variables.

3. Run descriptive statistics. The first step is to run descriptive statistics (frequency counts, percentages, mean, median, mode and measures of variability) on the entire sample. Look for any interesting trends or findings.

4. Run comparative analyses. This is where you look for differences among variables. First look for any differences based on the various demographic data you collected - gender, age, type of business, location and time of seminar, etc. Second look for differences based on previous experience e.g., are computer experienced participants more likely to set up their own Web site? Next, look at any differences among attitudinal factors. The purpose of this exercise is to identify the characteristics and profile of certain client groups.

5. Keep a realistic perspective. Watch your sample size when reaching any conclusions. This is important when you are doing group comparisons with small group numbers. The larger the number of respondents, the more stable will your data and results be. Statistical tests of significance can be useful. However, practical differences are more important than statistical differences.

6. Consider alternatives. Interpreting data is definitely an art that requires considerable experience. Trying to postulate the reasons behind certain results is even more risky. Keep an open mind regarding alternative explanations when interpreting your results.

5. Presenting Your Findings

Here are some suggestions regarding presenting your findings.

1. Tailor the findings to the situation. The comprehensiveness of the report will depend on your requirements. If you alone are going to be using the results, then the computer printouts and/or a writing of the percents next to the questionnaire items is sufficient.

However, if a sponsor or your corporation requires a report, then a more formal report may have to be prepared. This report may include the following data:

• Executive summary (if report is more than 10 pages long)
• Background to the training
• Goals and objectives
• Where and when the training was delivered
• A profile of the participants
• Results from the evaluation
• Conclusions and recommendations
• Appropriate supporting documents

Remember a well-written evaluation report can be used as a marketing tool. It shows to new potential clients that you are offering a quality product, and that you are professional enough to measure the results and use them for continuous improvement.

2. Simplify the statistics. Using a mean (average) or median is a lot easier to understand than listing frequencies by each category. Also, keep percentages to a whole number, not to 4 decimal places.

3. Use graphs. Graphs are easy to understand and read. Do not include too many variables per graph. Also graphs are useful to highlight differences between groups. One useful technique is to rank the graph from highest to lowest.

4. Use tables. Sometimes tables have to be used to present data. Keep percentages to whole numbers. Also, if the table presents a lot of data, it is often useful to order the results from high to low. That way the reader can see very quickly what factors are highest rated and which ones are at the bottom. If the sample sizes are small and you are providing percentages, be sure to list or indicate the numbers of respondents. This will reduce the risk of misinterpretation of the data.

5. Provide explanations. Even though you may think the findings in your graph or table are obvious, not everyone may be able to see them. Therefore, take time to write your interpretation down.

6. Include comments. The comments and responses to open-ended questions provide the substance to the numerical data. There are a number of ways the comments can be handled. One is to simply type them in. This is a useful strategy is the number of participants is small, or a very few participants took time to write comments. The other option is to do a content analysis of the comments. This provides a frequency distribution for a smaller number of categories.

Remember the whole purpose of collecting client feedback is to increase your profitability by improving your training products and services so that satisfied customers keep coming back and recommending you to their friends and co-workers. Therefore, writing up an evaluation report is not enough. The data must be used in an effective manner to ensure your products and services are competitive in the market place.