The principles and practice of the fire protection engineering exam serves as the method by which fire protection engineers become licensed professionals in the United States. Licensure as a professional engineer demonstrates that an engineer is competent to offer engineering services to the public, with the responsibility for the protection of public health, safety and welfare from fire and its effects. The magnitude of this responsibility is equal in intensity to the rigor associated with the identification of critical professional activities and knowledge/skills required for an individual to be considered competent to practice fire protection engineering, development of the exam specification, and the eventual assembly of each exam.
The National Council of Examiners for Engineering and Surveying (NCEES) is the organization that develops,administers and scores the examinations used for engineering and surveying licensure in the United states. It serves as the principal coordination and oversight body responsible for the professional activities and knowledge/skills identification process.
Between July and November 2010, the fire protection engineering exam specification underwent revision through a process called a Personal Activities and Knowledge/Skills (PAKS) Study. The PAKS study is intended to obtain information about the professional activities performed in a job and the knowledge and skills needed to adequately perform those professional activities. The output of the PAKS study is a test specification that serves as the framework for the examination that is used to evaluate an individuals minimal competency in a given professional field of practice. The PAKS process is required to be conducted every six to eight years, with the most recent study conducted in2002 to produce a specification that was first used in the assembly of the October 2004 fire protection engineering exam.
Development of the PAKS Study
The content of the PAKS study involves the technical support from a diverse team of professionally licensed subject matter experts (SMEs) serving on a task force and test specification committee, and survey participants. The diversity of the committees and survey respondents ensures that the exam specification accurately reflects the practice of fire protection engineering. The process commenced in July 2010 with 12 fire protection SMEs forming a task force to evaluate the professional activities and knowledge and skills important for an individual to demonstrate minimum competency in the field of fire protection engineering. The SMEs were diverse in gender, years of experience, field of practice, current professional responsibilities, type of employer and principal geographic area of practice.
The task force utilized the existing exam specification as the foundation for its analysis and evaluation of the professional activities and knowledge and skills The task force identified 34 professional activities considered critically important for a fire protection engineer to be able to perform. The group also identified 99 knowledge areas and skills related to those professional activities considered critical to evaluate through the examination process. After these professional activities and knowledge areas and skills were identified, they were grouped into specific domains. Defining the content domain is a critical component in establishing the content validity of the credential,and the content domains serve as the basis for a survey that was developed and deployed to validate the beliefs of the task force.
Survey of Licensed Professional Engineers
Prior to deploying the survey, 22 licensed fire protection engineers pilot tested the survey. The goal of the pilot test was clarity of wording, ease of use and comprehensiveness of content coverage. Upon receipt and adjudication of comments received from the pilot test participants, the survey was prepared and deployed.
This survey served to evaluate, through the response of a statistically significant number of licensed professional engineers, whether the professional activities and knowledge areas and skills identified by the task force represented those elements that practitioners felt were critical for determining the minimum competence of a licensed fire protection engineer. The survey was released in September 2010 and was open for five weeks. Only responses from licensed engineers were accepted. A total of 312 responses were utilized in the statistical analysis. The 312 response exceeded by 50 percent the minimum number of responses required for the data to be considered statistically significant.
More than 50% of the respondents passed the fire protection engineering exam to become licensed, spent more than 50% of their current time dedicated to hands-on practicing of fire protection engineering, and more than 50% of the respondents supervised or mentored other fire protection engineers. These statistics and the inclusion of this large number of diverse subject-matter experts enhance the confidence in the survey results.
An index of agreement (IOA) was developed to measure how well different subgroups of respondents agreed on which professional activities and knowledge areas and skills were important. The statistical analysis that evaluated the responses between subgroups concluded that a high IOA occurred between the following subgroups of respondents:
- years licensed as an engineer
- years in the fire protection field
- amount of time devoted to fire protection engineering
- job function
- area of primary practice
- supervise/mentor engineers with less than five years of experience
- highest engineering educational attainment
- degree in fire protection
Analysis of Survey Results
The test specification committee, composed of many of the same members of the task force, met in November 2010 to review the survey results and achieve a consensus on the professional activities and knowledge areas and skills considered important, and thus required to be included in the exam specification.
The first area of evaluation was for content coverage. The survey respondents were asked on a five-point scale, with 1 = very poorly and 5 = very well, whether the identified professional activity domains and the identified knowledge and skill domains adequately covered the required exam content. The means ranged from 3.66 to 4.07 for the professional activities and 3.74 to 4.08 for the knowledge areas and skills. This was considered sufficient to validate content of the survey.
The survey also requested respondents to rate the professional activities and knowledge areas and skills on a 5-point scale of importance with 0 = no importance and 5 = very important. Critical importance was determined by evaluating the mean importance rating. In this study, the midpoint between moderately important (2) and important (3), or a value of 2.50, was selected as the cut-off line above which professional activities and knowledge areas and skills were considered critically important. Those below 2.40 were determined to contain insufficient priority and were excluded from inclusion in the exam specification. Those in the range of 2.40 and 2.50 were considered on the borderline and were evaluated individually to determine whether the default cut line was appropriate. The test specification committees evaluation and adjudication of each mean importance rating was recorded as part of the specification development transcript.
The test specification committee concluded that those with borderline mean importance ratings, in fact, should be excluded from consideration in the test specification. The survey results and evaluation by the test specification committee decided that 27 of the 34 originally identified professional activities and 89 of the originally identified 99 knowledge areas and skills achieved high performance ratings and were required for the competent performance of a fire protection engineer. In addition to the quantitative responses, 125 narrative comments were received regarding the professional activities and 47 comments were received regarding knowledge areas and skills. The test specification committee adjudicated all of these narrative comments prior to achieving consensus on the final exam specification content.
The survey requested respondents to identify the percentage of each domain they believed should serve as the basis of the test specification. Without viewing the survey results, the test specification committee independently suggested the percentage of each domain that should be identified on the test specification.
The average of the test specification committee responses matched the survey responses with very good agreement. The test specification committee used the percentages to determine the number of items to be asked on an 80-item exam.
The final activity of the PAKS study was to link each of the knowledge domains to at least three professional activities. This process serves to affirm that each professional activity that was identified as being a critical component of the professional certification process was being tested by a specific knowledge area.
Finalization of the Exam Specification
The final exam specification was approved by the NCEES in February 2011. The new fire protection engineering exam specification will be used for the first time in the assembly of the October 2012 exam. The current and newly approved exam specification can be found in Table 1above (see page 40).
The complete specification for the next exam can be found at http://www.ncees.org/Exams/PE_exam.php.
The most significant changes in the new exam specification included the realignment of foam systems and water mist systems from water-based fire suppression systems to special hazard systems and the removal of an explicit reference to economic analysis. Foam protection systems, while water-based, were believed to be more appropriately associated with special hazard systems, and engineering economics was believed to be adequately covered on the Fundamentals of Engineering exam. While an exam item may still address economic analysis, it was determined not to explicitly identify this professional activity as one that required specific identification.
This new examination specification now serves as the foundation upon which future fire protection engineering exams will be assembled. The exam currently has 80 multiple response items. NCEES requires each area of professional practice to maintain a minimum of three exams worth of items, or 240 items, in an item bank. This item bank serves as the library from which items are withdrawn and used in the assembly of the annual exam. One reason for this minimum number of items is that items can only be used once every three years. This increases the reliability of the exam and reduces the possibility of compromise of exam items.
These items are created through item writing sessions held several times annually, and traditionally moderated by the Society of Fire protection Engineers staff. These sessions assemble registered fire engineers to develop exam items linked directly to a specification knowledge area or skill. After completion, each item goes through a quality control process by which it is validated for clarity and accuracy by two independent licensed professional engineers, and then finally approved for entry into the bank. Prior to being published in an exam, each item undergoes a final round of validation as each exam is pre-tested in the spring prior to the October exam. Pre-tests assemble a minimum of four licensed fire protection engineers to validate the clarity of each item and the accuracy of each response.
The fire protection engineering exam continues to perform statistically well, and the quality control process has provided the optimal result of not producing any item with more than one correct answer or where examinees must be given credit for any choice because of an issue with one or multiple item response choices. This performance underscores the quality of the item written by the professional fire protection engineering community, the quality control process, and the confidence that can be placed in those licensed fire protection engineers.
Anthony J. Militello is with the U.S. Navy.