Click here for a pdf file of this page.
American Society for Quality (ASQ) Toronto section has distinguished itself
as the section that witnesses maximum number of candidates appearing for the
certification exams. The number of CQE candidates has gone beyond any record
set any where else globally. With so much interest in ASQ certification, there are also numerous questions
about the exams from those interested to obtain one of these certifications.
Sometimes there are questions from the candidates who do not make it through
the examinations on the legitimacy of the examinations. To be specific: how
these exams are developed, and how the pass/fail point is determined. This article
is meant for all those who are interested to know about the development and
evaluation of ASQ certification exams and how to increase your success rate
in any such certification examination. An overview of the process of development and evaluation of any ASQ examination
is given in the process flow chart attached to this article. About ASQ certification exams in general ASQ certification examinations are different from any typical university or
college examination on a subject. They are designed to test an individual's
knowledge of a given body of knowledge to judge whether a certificate could
be awarded to that individual in that specific category of certified quality
professionals. These exams are not all that complex but are unique in the following: a) The body of knowledge for ASQ exams (like CQE) covers a wide variety of
topics that are drawn from several text books, and there is no prescribed course,
even though there are many agencies that offer courses.
In the case of a typical university subject exam, the volume of topics covered
is much more focused, and there is a prescribed course at the university with
term exams, exercises, and project work etc., that build clarity of concepts
and knowledge of applying those concepts through problem solving exercises and
discussions in the class and among classmates. The fact that this type of prescribed
learning is typically not available for ASQ examinations makes the examinations
a bit tough for many candidates.
b) Exam questions: The questions in the CQE certification exam are only multiple
choice questions and test mostly the conceptual clarity and general applied
knowledge of various subjects in the body of knowledge. The majority of these
questions really do not require detailed calculations. In fact over the years
the number of questions with very complex calculations or time-intensive problem
solving has been reduced significantly for the CQE certification exam. This
change is largely the result of technological innovation. Up to a few years
back the CQE exam was an all day test with two exams of 3 hours each (forenoon
and afternoon) and candidates had to do complex calculations.
c) The time factor: Many candidates complain that the number of questions in
ASQ examinations and the time allowed to do the exam limit the amount of time
you can spend on any question to less than two minutes. Perhaps a better way
to look at this challenge is to realize that a quality practitioner is quite
experienced with a variety of quality tools and techniques and is able to remember
those quality concepts including the nuances of applications of these concepts
and thus is able to answer many of the certification examination questions in
less than a minute. Of course some of the more complex questions may take as
much as 5 minutes or more to calculate or to make up your mind in choosing,
which of the options is the best fit when faced with choices that are quite
close. It is important to remember that the calculated amount of time per question
is only a mean, and, as any quality professional can tell you, that figure doesn't
factor in the range or variation from the mean.
d) Criteria for registering for an ASQ exam: A candidate registering for any
ASQ certification exam is required to have a combination of educational qualification
and practical experience in the field of quality.
The process of developing certification examination questions It is a long and costly process for ASQ that involves several hundred hours
of work by teams of certified professionals attending various sessions. The
key workshops involved in the exam development process are (a) Job Analysis
Workshop, (b) Test Specification Workshop, (c) Item Writing, (d) Item Review
(e) Exam Review. Participants for every session are to read details of the process
and relevant details as preparation before every workshop. (a) Job Analysis Workshop: This workshop of 12 Subject Matter Experts (SMEs)
results in setting the foundation for the Job Analysis Survey. The survey is
mailed to about 2000 SMEs in that field (irrespective of ASQ membership) to
determine what knowledge and skills are appropriate to test for the specific
certification exam. (b) Test Specification Workshop: The tasks and knowledge areas obtained from
job analysis are then assembled into an outline format with topics and categories
that logically group to form Body of Knowledge (BOK) for the specific certification
category. The SMEs then develop explanatory subtext that clearly describes the
particulars of each subtopic. The number of questions and the time limit for
taking the exam are also determined at this workshop. (c) Item Writing Workshop: In this workshop 24 SMEs who are trained in item
writing develop question items in small groups. Each item (assessment professionals
use this term instead of question) that is developed in this session must have
the following: - only 1 key (correct answer) that must have team agreement.
- A reference to support the key.
- A rationale for the item which explains what the item is testing and
why the key is correct and the other options are not.
- Classification to the most detailed level of the BOK. Item writing workshops typically produce 200-300 items over a two day period.
The items produced are considered `raw' and will be reviewed by another group
of SMEs at an Item Review Workshop. (d) Item Review Workshop: About 4 weeks before the meeting , the twelve SMEs
are sent the items to be reviewed in the workshop. They answer the items as
if they were candidates sitting for the exam. While they review the items they
are expected to make comments about them and submit those comments to the ASQ
test developer. The comments from all SMEs are then compiled into one document,
which is used for item review workshop. The SMEs attending this workshop are
trained in a variety of assessment issues. Their focus is to review the items
to ensure that there is only one key and that the item is clearly written and
accurate. They also confirm the reference and rationale for each item. once
all components of the item are verified the item is approved and can subsequently
be selected for use in an exam. (e) Exam-Review Workshop: once the raw items are reviewed and approved, the
ASQ test developer creates a draft exam in accordance with the test specifications,
and reviews it for overlap or cueing between items. The12 SMEs attending this
workshop take the exam just as any candidate would. As in the case of Item Review
discussed above, they are asked to comment on the items and send those comments
to ASQ Test Developer. At the workshop the members are trained in a variety
of assessment issues and then asked to focus on items identified as "problematic"
from the review prior to the workshop. once all the issues in the exam are resolved,
the committee approves the exam and it can then be used for test administration.
The process of determining a passing grade Whenever an exam is administered with a new body of knowledge, the exam is
subjected to a cut score process. A group of 12-14 certified professionals (other
than those who were involved in developing the questions or reviewing the exam)
is asked to rate each question in the new exam in terms of difficulty for a
minimally qualified examinee or candidate. In other words the team is asked
"What % of minimally qualified candidates for the exam will get this question
correct?"
The cut score study participants consider many factors such as the basic or
advanced level of knowledge required to know the right answer, any potential
errors a candidate can make because of the various choices given, the way the
question is presented (including the use of English language styles and words),
etc. Whenever members of the cut score panel show an expected performance score
that is more than 25 points apart from other panel's rating, the difficulty
factors for that question are debated and discussed until a consensus is reached
by the group. The final cut score developed by the cut score study panel is presented to
the ASQ Certification Board as the recommended percent or portion correct needed
to pass the test. See Appendix-B for a sample of the item statistics template
exam stats template used for discussions during the cut score study and a sample
summary table of what the ASQ Certification Board sees when determining the
cut score of a new body of knowledge. What is described above is a very brief description of the cut score process.
I was a member of the cut score study panel for the CQA exam administered in
June of this year, and this is the process we used to set the cut for that new
body of knowledge. It is the same process ASQ uses on all of its exams.
Note:
The procedures ASQ uses to develop tests and establish pass point are in accordance
with those described in the ANSI 17024 standard; these procedures are considered
accepted practices as outlined in the Standards for Educational and Psychological
Testing, which are developed by the American Educational Research Association,
(AERA), the American Psychological Association (APA), and the National Council
on Measurement in Education (NCME). What really works for a candidate to ensure passing any ASQ certification exam? Please note that points described below are my own personal recommendations
on how to prepare to take and pass an ASQ exam. This is also based on my experience
and the experience of others who shared them with me.
a) Go through the body of knowledge booklet for the exam and get an idea of
the topics.
b) Identify your strength and weakness in terms of your depth of knowledge of
the topic areas and check how comfortable you are answering some of the sample
questions.
c) Prepare for the exam by using a combination of the following (depending on
your background and knowledge comfort level):
(i) Self study using CQE primer, reference books from the CQE bibliography etc.
The CQE primer developed by the Quality Council of Indiana presents over 600
pages of subject material collected from several books. This primer also contains
a collection of 600 sample questions and answers that cold be used as practice
exams. (Note that that these primers are not developed by ASQ or in collaboration
with ASQ; but they published by private companies.) Reference books from the
Body of Knowledge (BOK) booklet for each certification (or through asq.org certification
website) offer much more in-depth review of specific subject areas. After reviewing
the body of knowledge booklet, the candidate will know which areas of the BOK
are challenging. It is well worth the investment to acquire specific textbooks
for these areas, because no primer or refresher course will give the necessary
depth of understanding possible through these sources. A good reference book
is also a good investment.
(ii) Taking college/university courses: For candidates who find complex some
topic areas (such as reliability engineering, or design of experiments) too
tough to understand by self study, it is good to try some specific courses on
that topic. Community colleges do have a number of such courses on many topics.
The need for such courses would depend on your background and expertise. For
candidates with good academic background and familiarity with statistical tools,
this need would be very minimal. However, keep in mind that by passing a set
of these college courses you cannot expect to just walk through the ASQ certification
exams. These courses really help to a large extent. But coursework alone is
not sufficient. (iii) Refresher courses: There are a few refresher courses offered by senior
quality professionals and just a few community colleges. once you have completed
the general preparation, it is a good idea to join any of the refresher courses
to get a sound, overall review and opportunity for discussion on some of the
more typical questions that are likely to show up on the exam. Many of the refresher
courses offer tips on formulas, definitions, etc., and can be quite helpful
to increase your confidence on tools that you don't use everyday in your work.
(iv) Mock exams: Using mock exams in the form of sample questions from various
sources (primers, ASQ information bulletin, refresher course material, etc.)
is a good way of exercising your brain and making it sharper for the actual
the exam. Through several such mock exams it is also possible to identify common
mistakes that you may make and develop strategies to guard against them during
the actual exam. How does our Toronto section perform in these ASQ exams? a) Our section performance in the certification exams is generally within the
± 5% of the global average. Sometimes our section performs better than
global average and some times a bit worse. A total of 180 * candidates sat for
the CQE exam this June in our section, and 67 passed; i.e., 37.2%. By comparison,
1,450 candidates sat for the exam globally and 610 of them passed; i.e., 42%.
Our best performance for CQE in the last few years was 62%. b) Several variables affecting the section performance in terms of % pass: - the candidates' background knowledge/educational background
- practical experience in the field of quality to be specific the Body of Knowledge
topics
- candidates' preparation for the exam
- performance during the long 5 hour exam with concentration and time management
skills
- strategy used for identifying correct answers when several choices compete
closely or where more in-depth knowledge is required. Our section is continually pursuing possible initiatives, within our limitations,
to help the candidates for getting certified. one thing we continue to do is
to make information on professional certifications available to all interested
parties through many ways, for example articles such as this, information posted
at our web-site, and through the information desk provided exclusively for ASQ
certifications at every monthly program meetings. K. C. Mathew P. Eng., C.Q.E., C.Q.A., Certified Quality Manager,
Certification Co-Chair, on behalf of the Certification Committee
ASQ, Toronto Section |