AUDIT QUESTIONAIRE FOR MR. WEISSMULLER
Describe the major changes that have transpired
since the current description of duties/responsibilities was written. Explain in great detail.
Furnished by incumbent
during audit on 18 Aug 2004.
Describe the major duties and responsibilities of
your position. Provide examples and
documentation.
Furnished by incumbent
during audit on 18 Aug 2004.
KNOWLEDGES
REQUIRED:
Describe the knowledge and skills required to
perform your work as described on the proposed core document. Give examples how they are used.
Normal Test Management Section duties require:
(Note: When viewed online: Knowledge items are in
blue. Level
of Contact are in Green, Guidelines are in Violet, Impact of work is in Red)
General
overview: The three major sections below represent aspects of my current work for
the Air Force in my current position.
The first knowledge section covers the
“core” responsibilities for any personnel psychologist who would fill this
position. The second knowledge section
covers “extended” responsibilities in direct support of
this Test Management Unit, but made necessary by decaying support
from historically responsive sources like a) data analysts – now reassigned to
Force Development, b) contracted programmers – leaving to take other jobs, c)
the only programmer with personnel research data processing experience being
sent away i.a.w. new AFPC 4-year rotational policies
or d) AF/DPX (formerly 3 positions, now 1) and the Air Force Human Resources
Laboratory (AFHRL formerly 30+ PhDs in Manpower & Personnel, now
NONE). The third knowledge section
addresses those “person in the job”
knowledge items that come from a blend of 30 years of Air Force research
and private sector expertise which causes diverse Air Force elements to seek
out his advice and input beyond the scope of the Test Management Unit.
SECTION
1: “Core Responsibilities”
1) Advanced
Psychometric Theory and Practices, Professional Standards for Statistics, Test
Evaluation & Research Methods:
These knowledge items are needed for
A) auditing, detecting and correcting
scoring anomalies in the internal Air Force Test
Management procedures (covered by AFI 36-2605
& AFPT 250 located at <http://www.afpc.randolph.af.mil/testing/>)
for handling exceptions to Testing Standards due to multiple, semi-parallel
test versions allocated in a strategic manner to combat test compromise. This software is also used by the Air Force Occupational Measurement Squadron (AFOMS – Test
Development Division) who develops the tests and monitors test
performance using item statistics produced by Bilog –
an Item-Response-Theory (IRT) program in the Rescore process. The incumbent uses data and statistics from
the AFOMS operation and has, on occasion, needed to go over there and help the
junior officers responsible for controlling and processing the sensitive test
data.
B) assisting the Air Force Office of Special Investigations in:
(1) serving as an
expert (test statistics) witness in court
martial prosecutions; explaining, in layman’s terms, statistical
concepts used in the validation of standardized testing programs and, in case
of test compromise, the probability of score histories occurring just by
chance.
(2) improving advanced
statistical methods to better target truly anomalous test histories indicative of test compromise leading to AFOSI
investigations, and
(3) serving on the AFPC-AFOSI-AFOMS
Test Compromise Watchdog team for recommending and promoting new, anti-compromise programs which PREVENT 1) the need to spend tens of thousands
of dollars to expedite the creation of replacement Air Force tests, 2) cheaters to assume managerial positions in
the Air Force, or 3) loss of confidence in the most advanced testing program on
the planet., and
C) using the
American Educational Research Association (AERA), American Psychological
Association (APA), and National Council on Measurement in Education (NCME)
joint 1999 Standards for educational and
psychological testing to validate new or proposed experimental
methods for USAF/DPX for pilot selection in
projects such as:
(1) the validation
of the Test of Basic Aviation Skills (TBAS) to replace
the BAT which has been used operationally for over ten years in evaluating and
screening USAF pilot candidates (ROTC, OTS,
USAFA, Active Duty Airmen pilot sources, etc.),
(2) serving as an advisor/participant on the HQ AETC Integrated Process
Team for Evaluating Pilot Selection Policy
(3) serving as
an advisor/participant in the AFPC Integrated
Process Team with AETC/SAS and AFOMS aimed at identifying the most
appropriate organizational element to host/support the OPERATIONAL and
CONTRACTING aspects of
the Pilot
Candidate Selection Method (PCSM, including BAT & TBAS –working modifications in conjunction with AETC Studies and
Analysis Section, AETC/SAS ) and the Air Force
Officer Qualifying Test (AFOQT) (as opposed to the research, validation,
and oversight aspects). This decision must balance practicality in combination
with the need for organizational commitment and sustainment
as well as strong links to core mission and continued test validity.
(4) evaluating the USAF
Pilot Interview proposed by HQ ROTC. Incumbent was selected to be the focal point
for discussions with HQ ROTC due to their past dissatisfaction with the use of
the AFOQT managed by AFPC/DPPWT. Based on very positive initial meetings with
the HQ ROTC sponsor and his contractor, the incumbent was given a scholarship
to be trained in the proprietary USAF Pilot Interview Analyst program at
Maxwell AFB, with HQ ROTC picking up all the expenses. While TDY to Maxwell AFB, between class times
the incumbent made personal courtsey visits to all
the organizations and offices which helped in experimental data collection for
the new AFOQT Form S test. During these
visits he back briefed them on how much their efforts will impact on the Air
Force in the coming years. These
supporters were provided the links to the web pages that documented this
research, its findings, and the benefits to the Air Force. <http://www.icodap.org/papers/Optech/Optech.htm>
These contacts also included the incoming DO for HQ
ROTC who asked the incumbent to research some additional issues to help in
evaluating an existing placement dilemma.
The incumbent provided the DO the requested information from historical
sources and laid down plans for updating that research (identifying and remediating people with skills but who do not “test well.”)
(5) reviewing similar
and possibly “purple” [DoD-wide] testing efforts within our counterpart
agencies in the US (Army, Navy, Coast Guard, FAA) and allied nation DoDs, and
(6) presenting the Air
Force’s Testing Program contribution in the subject of officer and pilot
selection to international conferences covering both logistical operations and advanced
research. AF/DPX and AFPC are sharing in
the cost of the TDY to serve as AF/DP representative to the International
Military Testing Association in Brussels, Belgium 25-29 Oct. AFPC/DPPPWT is on the steering committee and
assists in planning and establishing the overall focus and direction of this
international professional testing organization that promotes cross-flow and
exchange of research and operational practices.
SECTION 2: “Extended”
Responsibilities in Direct support of the AF Test Management Section
Note:
The personnel psychologist positions in AFPC/DPPPWT have historically
served as the liaison between the professional test and research function at
the Air Force Research Laboratory (AFRL) and the operational testing programs;
the operational arm for Air Force testing policy at Air Staff; and vital
coordination link with other relevant Air Force, DoD and other Service, other
Government, and private sector personnel researchers, psychometricians,
and operational testing program managers.
The personnel psychologist positions have ensured that the operational
testing programs conform to the professional standards for personnel selection,
classification, promotion, and proficiency testing. These positions coordinated with, met with,
worked jointly with, etc Air Staff, researchers, and all of the relevant
players required to develop, administer, and maintain
valid tests. When the Air Force
eliminated the internal research function and downsized the testing policy
function, these related functions were still required to have a legally defensible
and professionally viable fair and equitable testing program. In addition, much of the responsibility for
explaining the technical aspects of testing and for responding to senior
leadership inquiries shifted to AFPC/DPPPWT.
Through management of test research and development contract,s we keep the operational testing program current
and prepare for future modifications and improvements (for example, automated
testing). Just as the Air Force cannot
maintain a viable weapon systems without R&D and maintenance, neither can
we manage an operational testing program w/o these elements. Functions that were handled by a l00+ member organization which included many GS-14 and
GS-15 personnel psychologists have now become the responsibility of AFPC/DPPPWT. They are not optional and nice to do duties
but are vital to viability of the operational program and the well-being of the
Air Force.
2) Systems analysis,
contracting, and auditing skills (in combination with 1) above. These knowledge/skill items come into the
picture when interacting with state-of-the-art psychometricians and other personnel research experts
charged with implementing AF/DPX initiated/funded efforts such as:
A) add, score, deploy and exploit an
experimental personality assessment section to the Air Force Officer Qualifying
Test [AFOQT - used to screen ALL USAF officer
applicants], and recommend research to exploit this new TYPE of test
(personality) with sensitivity to professional
standards & military differences from guidelines in the civilian sector,
B) adjust USAF operations and research
programs to use a new
DoD-mandated norming standard on the
Armed Services Vocational Aptitude Battery [ASVAB -
used to screen ALL enlisted applicants (USAF, Army,
Navy, Marines, etc)] – and recommend research to exploit new ASVAB
subtests for modified USAF selection and classification
applications,
C) provide Contracting
Officer’s Technical Representative (COTR) services in conjunction with
my supervisor, Mr. Kenneth Schwartz, for both testing
initiatives above and related personnel research initiatives funded by AF/DPX such
as the
1) Revalidation of the Weighted Airman
Promotion System [WAPS used for ALL enlisted promotions
to grades E-5, E-6, and E-7]
2) Revalidation of the Senior NCO Promotion
System (SNCOPP used for ALL enlisted promotions to
grades E-8 and E-9],
3) Classification Decision Support System [CDSS- an Air Staff system for modeling the impact of
realignments of aptitude (ASVAB) entry requirements across ALL Air Force
Specialties (AFSs)] ,
and
4) Evaluation of the Impact of replacing the accepted 30-year standard of four Air
Force aptitude area selection composites [Mechanical, Administrative, General,
and Electronic – MAGE used in the classification/placement of all incoming and
cross-training enlisted personnel] with restructured composites (changing from
four to five, six or seven) taking advantage of new subtests in the
ASVAB and recommend new USAF selection/classification
systems to better match Air Force’s personnel development pipeline
structure to the actual spectrum of requirements in the field along with a
realigned training strategy to meet required job performance proficiency
levels. (“Right Person,
5) Validation of contractually obligated
“Government Furnished Data.” Initial data turned over the contractor from the current HRRD
included erroneous/malformatted data from both AFPC
and non-AFPC data sources. In
particular, the “audit” functions within the HRRD have not been supported since
1998 and changes in data locations or coding standards were not properly
tracked nor recorded. The AFPC/DPD
programmer resolved part of the reformatting issues on his own. The incumbent here
provided access to alternative data sources for personnel data and tracked
coding problems back to an OMB-mandated standard for all federal agencies
published in the Federal Register in 1997. This OMB standard covered the coding of
“Race” in all federal systems. The
incumbent tracked the history of how this was implemented within APFC (as of
Feb 2002) and did the computer runs to provide the contractor with correct,
consistent data for contractually required analysis. A related ancillary data function also arose when the most recent promotion cycle for 04E5 came to screeching
halt. Both AFPC and AFOMS (the
office responsible for test development and producing “final score keys.”)
argued that they had not changed their processes, however, 137 people showed up
with only 99 items for a 100 item test.
The incumbent reviewed the process and determined that the RACE code,
now standard in AFPC’s MILPDS system had a coding
error – resulting from the personnel entering in 2002 who just now became
eligible for WAPS testing to E-4. When
the race variable was corrected, the 04E5 promotion cycle proceeded without
problems. The
new race coding structure also presents new challenges for upcoming AF/DPX
contracts evaluating adverse impact due to DoD changes
in ASVAB subtests and Air Force consideration of new possible
selection/classification composites.
D) Although the
Test Management section is responsible for the controlled test materials, the
logistics are administered by AFOMS.
For this reason, the incumbent lead a re-design and will code a
web-based application to track the Test Material
Inventory (TMI) which is now managed as over 1000 separate spreadsheets with
manual, semi-annual updates.
Design will expand the currently used “Order System” and integrate
inputs from the printer, the Test Control Officers (TCOs),
the AFPC Test Scanning Section, and the AFOMS inventory manager to produce a
secure system with a real-time destruction certification logging followed
requests but with a human filter to ensure the integrity and paper-trail on
“Controlled Test Material” items.
SECTION 3: “Person in the Job” Knowledge Items
3) Other duties are assigned as a result of “the person in the job” with a diverse and unique skill set combined with a comprehensive
corporate knowledge of relevant, past Air Force research findings. The incumbent is called upon to make
substantive contributions to the important, critical and time-sensitive
missions of the Test Management Section, the Enlisted
Promotions and Testing Branch, the Promotion, Evaluation and Recognition
Division, the Directorate of Personnel Programs, the Air Force Personnel
Center, AFPOA and HQ USAF/DPX, the Air Force Occupational Measurement
Squadron, Recruiting Service, and other personnel researchers in and out of the
Air Force.
===== Background Note of Skill Set and
Corporate Knowledge Available and Drawn Upon in this position===
Knowledge used in these efforts was gained
A)
in six years of post-High School education (BA
in Mathematics with Minor in Computer Science/Systems Analysis, and 30+ hours
towards PhD in Educational Psychology – Quantitative
Methods).
B) and is augmented by thirty years of practical Air Force Personnel Research
experience primarily with the Air Force Human Resources Laboratory (AFHRL) and
its successor organizations in human resource-related areas (1971-2001) serving
as
1) an enlisted programmer supporting personnel research in
occupational analysis, training, testing, and manpower
projections/authorizations,
2) a systems analyst (military/civil service),
3) a principal investigator (occupational analysis/training/testing),
4) a transfer of technology representative for the Air Force
Human Resources Laboratory (AFHRL) for domestic and foreign distributions,
5) a government Contracting Officer’s Technical Representative
(COTR),
6) as a GS-12, step 4, selected as an Air Force Comptroller
cadre-member for the career progression program
7) ADP Specialist
(contractor) for designing experimental (6.1 funded) Air Force Personnel
testing,
8) Site Manager
(contractor) managing a 5-year $5.4 million contract for Human Resources
research,
9) Senior
Scientist/Project manager for 30-40 personnel research projects (occupational
analysis, testing & training) under Air Force contracts (1984-2003)
10) Occupational
Analysis/Business Process Reengineering Consultant to other federal agencies
and international clients, including the US Army, US Coast Guard, IRS, NSA,
FBI, AT&T, Human Resources Research Organization (HumRRO), Ontario Hydro
Electric (Canada), BHP (Steel Industry in Australia), Phelps Mining (global),
and world-wide studies for the United Nations (for HQs in Kuala Lumpur, New
York, & Vienna).
11) A Corporate
Founder and Senior Executive: a)
Sensible Systems (SSI, 1982-2003), b) Institute for Job and Occupational
Analysis (IJOA, non-profit, 1993-2001), c) Innovation Center for Occupational
Data Applications & Practices (ICODAP, non-profit, 2002-2004), and d) Medical Information Datamining
Applications & Practices Institute (MIDAPI, non-profit, 2002-2003).
12) an active
member of Professional Organizations such as the San Antonio Chapter of the
International Society for Performance Improvement (ISPI, founded in 1962 here
in San Antonio, Past Chapter President & current Web Master ) –Formerly a
member of the premier computer professionals organization: ACM – Association for
Computing Machinery.
13) Web Master for
several organizations such as Sensible Systems (www.codap.com),
ICODAP (www.icodap.org), Past Web Master for
other local chapters of non-profits such as the American Statistical
Association (ASA), the American Society for Training
Development (ASTD). Web master for the http://www.icodap.org/papers web page with the
largest collection of “free access” Air Force personnel research (occupational
analysis, personnel, manpower & testing) outside of the Air Force.
=========== End Background Note ===========
====
Examples of Extended Use in Current
Position/Organization/Air Force/Professional Community
(Note: Even though defined as extended
use many of these functions are critical to the viability, validity, and functioning
of the operational testing program)=====
TEST MANAGEMENT SECTION
Mr.
Kenneth Schwartz, Chief, Air Force Testing, recognized the incumbent’s
potential value to this SECTION due to his knowledge of test theory, personnel
research, computer systems and the Air Force.
1)
Mr. Schwartz reassigned the computer “rescore”
process from the unit’s Master Sergeant programmer/analyst to this new GS-11
personnel research psychologist position.
Mr. Schwartz asked the incumbent to thoroughly audit the “Rescore Process”
to explain anomalous results, rather than just assuming the case processing
duties. The incumbent’s background in psychometric theory
combined with a degree in mathematics, the ability to read the computer source
code delivered by the contractor, and systems analysis/auditing/business
process reengineering skills helped him to first, set up computer scripts to
reduce normal processing time from 5 work days down to four hours. Note:
This function is vital to operational testing program and would have
historically been supported by AFRL personnel researchers and psychometricians.
Elimination of that mission by AFRL and limited research contract
dollars does not eliminate the need to support this function. Incumbent is able to support vital function
and prevent mission failure)
2)
Mr. Schwartz,
performing Contracting Officer’s Technical Representative services for USAF/DPX
contracts, had to coordinate contractually promised “Government Furnished Data
(GFD)” from the Human Resource Research Database (HRRD) maintained at the Air
Force Personnel Center (AFPC/DPD).
His request for data uncovered the fact the despite a Memorandum of
Agreement (in force 1999 -30 Sep 2004) was not being supported and the
requested data was not available. His
efforts to secure the data also led to many other “disconnects” in the Air
Force’s ability to provide historical human resource data to contractors. The use of contractors has replaced the
function previously provided by the Air Force Human Resources Laboratory which closed
down in 1999. The incumbent stepped in
and assisted in integrating data sets obtained by the contractor and others
into data bases meeting the requirements for the GFD on the contract to
revalidate the promotion systems used for all enlisted personnel. Problems remain on other contracts and the
incumbent is working with AFPC/DPD personnel to solve the problems. Update 31 August
2004: The AFPC/DPD programmer was one of
the original personnel sent over from AFHRL (AL) in 1998/1999 to support the HRRD. At 10:58 AM this morning his requested
deferment (to remain at AFPC and continue this function) was formally
denied. He will be scheduled to leave
shortly. As he was the only person
available to access the HRRD in response to contractually obligated “Government
Furnished Data,” the incumbent will perform those responsibilities until the
HRRD function is re-established by the six AFPC Directors.
3) Mr.
Schwartz raised the problem of the non-functional HRRD to the AFPC level. This resulted in a
meeting with not only our Director, Col Maurmann
(AFPC/DPP), but with all six top Directors of AFPC in which Mr. Schwartz was
charged with the responsibility to develop a requirements statement to support
future HRRD operations. To
provide Mr. Schwartz (a Personnel Test Psychologist) with the details of a
broad spectrum human resources research data system, the incumbent contacted
current and retired Air Force researchers, constructed an Internet-based survey
of Air Force Research Needs, Necessary Data Files, Essential Elements, and
Specialized Software needed. This
Internet survey was sent to researchers and research requestors he identified
throughout the country and resulted in a comprehensive overview of Air Force
needs by the people who would know both analysis needs and necessary support
elements. Results may be found at: http://www.icodap.org/040510/results_page.html. With these results in
hand, Mr. Schwartz conferred with USAF/DPX who then asked for a Concept of
Operations (CONOPS) be prepared for the HRRD for action at the Air Staff level. The incumbent worked closely with Mr.
Schwartz in drafting the CONOPS and provided key details and insights as to how
AFHRL used to service Air Force-wide requests.
This CONOPS is currently in coordination within AFPC and AF/DPX.
4)
The incumbent has an extensive collection of
AFHRL personnel research reports covering topics directly related to
initiatives within this section. The
incumbent routinely (at home, on his own time) scanned relevant Technical
Reports (TRs) and loaded them to his non-profit web site. The non-profit, “ICODAP” was incorporated
during two years of unemployment to share “unrestricted” Air Force research
with industrial/organizational (I/O) psychology graduate students. < http://www.icodap.org/>. USAF/DPX has hired a contractor to scan the
remaining 1000+ Technical Reports in his collection to provide both Air Force
and graduate students FREE access to the high quality human resource research
performed by the Air Force since 1954.
Incumbent is working with key Air Force contacts to have “restricted
distribution” classifications removed from old “index” reports (1981-1995) to
permit improved access with online (rather than hardcopy only) search methods.
For tracking progress of this work, see the topic labeled “Shipment Control
Sheets” on web page: http://www.icodap.org/papers/AFHRL/DataFiles.html. Update 31 Aug 2004 –
on 20 August the contractor requested all TRs by end of fiscal year. A push was made to expedite the inventory
process by incumbent at home, on his own time. The incumbent provided, from his personal
files, 67,000 pages from over 1040 Technical Reports that represented over 36%
of the personnel research reports indexed and published between 1953 and 2001
by the Air Force Human Resources Laboratory.
The incumbent turned over the last of these materials this morning to
the Air Force. The project is now
entirely in AF hands.
5)
Portions of this site are now used for
within-section projects, communicating with
1) contractors, 2) other Air
Force offices, 3) other services (Army, Navy) and even 4) foreign DoD (United
Kingdom). These papers are at the
following “open” web link: http://www.icodap.org/papers/ For example - based on a 30 August meeting
with the Navy office developing new pilot and navigator training and selection
instruments (ASTB), the incumbent provided links to existing key web pages
documenting scientific details of our officer and pilot selection research http://www.icodap.org/papers/#Organizational%20Issues%20&%20Papers
6)
They
expressed an interest in job analysis techniques as well as Internet survey
tools. With one
tailored web page, the incumbent, provided the Navy
with quick access to responses to their issues and may save them time and money
by using available Air Force tools (also previously used by Navy pilots) rather
than developing and validating a new tool from scratch. <http://www.icodap.org/papers/SurveyTech.htm
>,
7) In
order to expedite communication and collaboration on section work, the
incumbent, as a skilled Web Master, used his non-profit web. This site was used to coordinate access to
reference reports and large, non-sensitive files (too big for contractor’s
email system) and motivate OUR participants in various research endeavors
through this office such as:
a) CMSAF
b)
Internet
Exit Surveys for WAPS Panel Members, shown under “results page” http://www.icodap.org/040510/results_page.html. to communicate survey
results to contractors writing the final report,
c) the “Last minute” effort to upgrade the
WAPS project http://www.icodap.org/040716/index.htm,
d) as a member of Col Lee’s development team,
the incumbent established a web link to communicate an “empty” prototype access
database to a contractor working on the pilot
Enterprise Information Management (EIM)
System for AFPC/DPPP (Col Lee) – an evaluation project for an AFPC-wide EIM in
the future: http://www.icodap.org/040823/index.html
,
i)
5and
e)
Coordinate
priorities using an online (restricted), up-to-date list of the incumbent’s
CURRENT Projects with Hot items indicated: http://www.icodap.org/papers/restricted/USAFonly/JJWs-Current-Projects.htm
8) .In
May 2003, an $94,000 NCS Pearson 5000i optical scanner
was delivered to replace an aging Opscan 10
scanner. These scanners are used to scan
and automate ALL responses for the Air Force Testing Program on all Air Force
Officer Qualifying Tests (AFOQT, 12-16,000/year), and all Enlisted Promotion
Tests (Specialty Knowledge Tests – SKTs, Promotion Fitness Exams – PFEs, USAF Supervisory Exam – USAFSE, about
214,000/year). Due to cutbacks during
the procurement, the support needed to program the new scanner was cut and
programmed to be accomplished in-house. Then internal AFPC reorganizations
reprioritized programming support and the new scanner has been left, unused
since installation with a $900/month maintenance contract. The in-house/contractor programming project
remained on hold for some time. This
DPPPWT job incumbent fielded two successful applications shown below. Finally, when the AFPC/DPD-contractor
programming began, he was decided that the scanner software was written in an
off-brand version of C++ which was not common in AFPC. The vendor supplied code was replaced (on two
occasions) with newer versions of the Microsoft C++ compilers including the .NET
version by AFPC/DPD personnel. The first
(and only) two successful applications on the 5000i were rendered
unusable. It was later learned that the
manufacturer does NOT SUPPORT interface to those newer compilers – the original
software HAD to be used. The AFPC/DPD
“in-house contractor” accepted another job and the project was
picked up shortly by a senior programmer who was very shortly deployed to a
NATO site in
{Background Note: In 1975-76, the incumbent was in charge of
the programming unit setting up optical scanners for AFHRL and the USAF
Occupational Measurement Center for the 75,000+ USAF Job Inventories automated
annually. (See Weissmuller,
J.J., & Kauffman, J.B. (1976, October). "Data
processing problems in collecting job survey information." Proceedings of the 8th Annual Conference of
the Military Testing Association (pp. 783-796),
Since arriving in this job on 3 November,
the AFPC/DPWWWT incumbent has:
a)
programmed the new
5000i scanner to score both PRE- and POST- tests for our (AFPC/DPPPWT) new Test
Control Officer (TCO) training programs – producing
scored answer sheets to the instructor and students within 10 minutes of test
close. The incumbent also
established spreadsheets to demonstrate the net
improvements of students due to the course and formatted this information to
expedite reporting the success of the new program up the chain.
b) Programmed the new 5000i scanner to report Exit Survey Results from three panels of 10-14 Chief
Master Sergeants assembled at AFPC to revalidate the entire enlisted promotion
system (both WAPS & SNCOPP).
The last group completed their customized Exit Survey at noon on
Thursday, and results of all three panels were briefed
with final slides to CMSAF
c) Recovered the 5000i vendor-supplied code
twice after it had been erased by AFPC/DPD in-house programming technicians.
d) Most substantially, a new form of the AFOQT
is READY to be deployed. After a year
more than a year of Research and Development and various other logistical
issues, the new AFOQT, Form “S” sits poised, ready to use. All is in place with 20,100 Form S answer
sheets, 11,900 Test Booklets, and 1500 Administrator Handbooks in inventory in
the warehouse awaiting distribution orders.
This new form (Form S) reduces test administration time by an hour
(making it LESS than ½ day) and adds new measures for personality profiles – of
key interest in emerging personnel research studies.
Every day that this
roll-out is delayed is an opportunity missed for a) Air Force economy
(saves 1 hour administration time), b) the Air Force Officer pipeline (permits
TWO Officer test sessions per day instead of one) and c) the Air Force research
program (new personality data that could be used for Officership
studies, Retention studies, and perhaps even Force Development for career
broadening issues. This new form impacts on the scanner requirement
because it needs 1) new form definition to the scanner including audits and
user exits, 2) new set of tests and changed lengths, and 3) new scoring logic
with file extracts for a) Master File records, b) Immediate Posting on the Web,
and Export to the Pilot Candidate Selection Method (PCSM) group at AETC/SAS for
screening pilot and navigator candidates.
The incumbent has already programmed the scanner to properly capture all
responses on the new Form S and is currently writing a single scoring program
which will produce all needed file extracts in one processing step. This new approach bypasses the need to export
scan files to other sections and await scoring results. This means that the AFOQT Form S results will
be posted to the Web faster than before with less opportunity for test
compromise or human error. With this
program complete, the Test Management Section will be able to cancel a contract
request for outside programming and, instead, have the
resources redirected to meet other programming needs to reach compliance with
new DoD standards for ASVAB scoring which began on 1
July 2004. This program is expected to come online by 15 September and
permit an immediate roll-out of the full AFOQT Form S battery world-wide.
9)
CMSgt Mark
Billingsley, Chief Enlisted Promotions & Testing Branch now routinely makes
direct requests to the incumbent in a number of areas. These areas include a) high-visibility, short
turn-around , ad hoc data analyses, b) responses to members or Board of
Corrections to Military Records (BCMR) inquiries where testing scores or the
mathematics of the promotion system are at issue, and c) tracking sources of
web-based compromises of the promotion system or other inherently “computer”
services. Update: On 20 August 2004, Chief Billingsley presented the incumbent
with a Certificate of Achievement with a $250 award.
a) The high-visibility inquiries require quick, reliable, contextual answers to queries from
Air Staff, CMSAF Murray, and the new AFPC Commander General Przybyslawski
and his staff, like CMSgt
McCoy. Although Chief Billingsley
used to rely on the system analysts in the section and data analysts in
AFPC/DPA for data analyses. Due to
enforced reassignment policies and reorganization within AFPC, formerly
reliable sources are less and less available, if at all. In addition, the incumbents’ analysis results
provide not only data but also briefing charts, and interpretations of why the
Air Force’s processes result in the trends shown. These topics have included:
i)
Research for
Chief prior to Air Force Times interview on “Is the EPR broke?”
ii) Are airmen working new systems (FA-22 and
CV-22) disadvantaged in Promotion Testing
iii) Does being assigned as a 3S0xx at AFPC help
or hurt promotions in the E-7 & E-8 ranks
iv) How spread out is the workload on Test
Control Officers across each testing cycle?
b) The response to members has been
accomplished by writing the response directly from the Personnel Research
Psychologist to the member with Chief’s endorsement. In one BCMR case it required explaining the
logic and inherent fairness of the mathematics in the supplemental promotion
process – how he could score high on the board scores and still miss promotion
by 0.7 of a point out of a possible score of 450. In a second example, a
SSgt read a quote from Chief Billingsley in the Air Force Times and wrote the
Chief to tell him he was wrong. The
Chief said, “if you study hard, you will get
promoted.” The SSgt said he studied but
he didn’t get promoted because “he doesn’t test well.” I review his entire test history from his
ASVAB scores through all his promotion tests.
All the test history showed that he was “in the middle of the pack” in
terms of ability. He was in a computer
career field which was full of fairly bright people. I explained to him that the Chief was
referring to WAPS in which “testing” is a very important component. I further explained that promotion rates are
returning to normal after several years of very high rates and that the effects
of his study was evident, but he was short of promotion by the points awarded
for two more years in the service. With
his two years in grade, and the average person promoted had four years in
grade, he was right in line – not every test can result in a promotion. The Chief concurred with that response.
c) Inherently computer talents are called upon
in a number of ways. When the Promotion
List for 04E6/E7 was compromised and published to several web sites, the
incumbent used his knowledge of web sites to help the Chief and the OSI to
identify names and phone numbers of web masters and the identity of those who
had handled the compromised promotion list.
Other examples include creating CD-ROM video disks from videotapes (of
Chief Murray endorsing the Senior NCO Promotion Program), using mailmerge to produce custom correction labels for 4860
rater cards in the SNCOPP Revalidation Panel, creating/conducting WAPS
follow-up surveys over the Internet using Microsoft FrontPage, or training
staff on scanning documents and using Adobe Acrobat Professional for producing
online copies of Air Force Instructions in text-searchable format.
10) Colonel Nancy Lee, Chief,
Promotion, Evaluation and Recognition Division,
selected the incumbent to serve on the development team for a division-wide
pilot of an
11) Colonel Steven Maurmann,
Chief, Directorate of Personnel Programs,
recognized the incumbent’s diverse contributions, including his work on helping
the Air Force to transition to use the new DoD normed
scores on the Armed Services Vocational Aptitude Battery, leading the AFPC
validation of the USAF Pilot Candidate Interview process, serving on the AETC
Integrated Process Team evaluation Pilot Selection Methods., serving on the
AFOQT Integrated Process Team and other accomplishments.
12)
Ms
Tina Strickland, AFPOA and HQ USAF/DPX, is
aware of the incumbent’s talents and history.
In addition to conferring with the incumbent on routine testing matters,
like the new DoD ASVAB norms, she has called upon him
to use his knowledge of Air Force personnel research history. In one such case, she called on the incumbent
to aid the Air Force in responding to inquiries from DoD. In preparation for a new round of research, DoD requested existing data on a “Job Performance
Measurement” (JPM) project they funded in the late 1980s. The Army, Navy, and Marines reported that the
project was done so long ago that data are no longer available. Ms Strickland called the incumbent to ask if
he knew of any sources for this data from Air Force studies. The incumbent
reported that he suspected that the data were PROBABLY on one of the 400
CD-ROMs created when AFHRL closed its doors in 1999. The CD-ROMs were stored, without any index,
in several large boxes under a printer in the AFPC/DPD computer room. Some CD-ROMS had just a four digit study number
as a label. The incumbent researched and
found an automated index that covered about 1/3 of the CD-ROM, linking the
study numbers to study descriptions. The
incumbent was aided by others in the unit to automate any descriptions found on
another 1/3 of CD-ROMs. The incumbent merged all available titles into a Master Excel
spreadsheet, to become part of the Human Resources Research Database
(HRRD). This index identified six
projects on JPM and the data was indexed and the list forwarded to AF/DPX to convey
to DoD.
13) Ms Cynthia Luster,
Air Force Occupational Measurement Squadron (AFOMS – Occupational Survey
Division), calls upon the incumbent to provide
recommendations and explanations to their contractors and staff responsible for
creating the multi-year, $7 million software system at AFOMS known as CAROMS. While parts of CAROMS service routine
administrative tasks, a large portion of the system will be dedicated to
replacing the Comprehensive Occupational Data Analysis Programs (CODAP) system.
=== Background Note ===
The incumbent worked as a programmer on
CODAP from 1971 to 1974. From 1975 to
1981, the incumbent was in charge of CODAP development and the design of
occupational analysis-related projects for AFHRL – and that system supported
AFOMS until they purchased their own mainframe computer. As a contractor, the incumbent headed every
conversion effort of CODAP used by the Air Force, Army, and Navy. The incumbent performed dozens of CODAP
installations including NSA, University of Texas at Austin, Florida State
University, Ontario Hydro Electric, San Antonio City Public Service, etc.
=== End Background Note ===
Because of his
value to the large Air Force contract, they requested permission to use the
incumbent’s consulting services during the day and Mr. Ken Schwartz, his
supervisor agreed.
14) Mr. Paul DiTullio, Recruit Service Studies and Analysis, calls in
reference to the source of or analysis of statistical data he has been
provided. In one example he was given
population statistics for accessions 1995 to 2003 and he noticed some
anomalies. He called the incumbent who
brought up similar statistical data for 1981 to 1990 and demonstrated that the
pattern was consistent over decades.
15)
While
there is no central office for personnel research in the Air Force, many
researchers are still active with projects within their area of expertise. In one such example,
Dr. Thomas Carretta performs research related to
pilot selection and related studies like the Air Force’s strategies for future
use of remotely piloted vehicles.
Because of Government Furnished Data in other contracts, AFPC/DPPPWT has
access to statistical training records relevant to diverse researchers
needs. Permission has been granted for
the incumbent to provide Dr. Carretta with data extracts
relevant to his on-going research project and preclude
separate requests to an already overtaxed Human Resources Research Database
(HRRD).
SUPERVISORY
CONTROLS:
How is your work assigned? What is your responsibility in carrying out
the work? How is your work
reviewed?
The Supervisor provides a broad scope of the issues to be addressed and any concerns about possible policy impacts. The incumbent outlines an approach and discusses recommended actions with the Supervisor. Supervisor attends meetings with incumbent and action officers to achieve mutually agreed upon courses of action. Supervisor reviews final drafts of reports to evaluate recommended deviations from civilian psychometric guidelines and compliance with emerging Air Force needs and standards.
GUIDELINES:
What guidelines do you use in performing your
work? How much judgment do you exercise
in using these guidelines? Provide
examples.
Guidelines are discussed in the Knowledge section and are colored “plum” for easy identification. The core
responsibility of this job is to interpret, evaluate and approve or disapprove
deviations from published Air Force Testing
policy. Sound professional judgment is
used to amend
and coordinate updates to Air Force policy in AFIs governing Air Force Testing. These AFIs are AFI 36-2605 &
AFPT 250 located at http://www.afpc.randolph.af.mil/testing/
The incumbent
must evaluate the compliance of proposed Air Force Tests with the American
Educational Research Association (AERA), American Psychological Association
(APA), and National Council on Measurement in Education (NCME) joint 1999 Standards for educational and
psychological testing. The
incumbent must use professional judgment, in conjunction with the Supervisor to
recommend and defend deviations from civilian standards for special needs and
circumstances by the
COMPLEXITY:
Describe the most complex
work you perform. Provide examples and
documentation.
Breadth
The primary source of complexity arises from switching between diverse professional domains in order to achieve a single required result. Auditing the “Rescore” process for example required psychometric knowledge of “Item Response Theory”, statistical knowledge of “sampling effects”, prior knowledge of the language skills of various Air Force Specialties, knowledge of Air Force selection/classification policy, computer knowledge to read and understand the actual functioning of “source code” delivered by the contractor three years prior, knowledge of Air Force data bases to determine promotion impacts back to September 2002, and professional sensitivity in recommending proposed remediations. Again, with the elimination of other psychometric, testing, and research expertise in the Air Force, the demands for breadth of knowledge and skills in performing the job is great.
Depth
The second source of complexity has to do with the depth of understanding required to evaluate pragmatically devised proposals for Air Force tests. This evaluation must address 1) the psychometric validity, 2) “fairness” to the members, and 3) incremental validity (cost/benefit) to the Air Force selection process. When tests like the “USAF Pilot Candidate Interview” process is developed by an operating agency (in this case HQ ROTC) using non-traditional methodologies many issues arise. Because HQ ROTC is happy with preliminary results and suggests Air Force-wide implementation, the issues are political, logistical, and professionally demanding. Not only do the traditional “test validation” concerns exist, but the non-traditional way in which this instrument was devised makes it hard to interpret an appropriate validation methodology. Moreover, as the contractor was not familiar with the military, more scrutiny had to be given to possible “defenses” if challenged in court.
Impact
Attention to detail is
imperative at all times. These responsibilities take place in the arena of the Air
Force Testing Program that covers both the officer and enlisted force. Confidence in the fairness and validity of
this program is essential. Failure to
proactively preempt challenges could lead to lost of confidence by members,
adversely impact on reenlistment rates, or result in the media calling into
question the Air Force’s commitment to a quality force.
Describe the nature and variety of tasks, methods, functions, projects,
or programs carried out in the position being described.
Some information
furnished by incumbent during audit on 18 Aug 2004. Any additional documentation/information to
facilitate in classifying the position is appreciated.
Please refer to Section 1
above for a comprehensive review of the diversity of functions.
Describe how you identify what needs to be done in performing your
work.
After the area of concern
has been identified by the Supervisor, the incumbent uses professional judgment
to determine the most appropriate approach consistent with industry standards
for personnel research as modified by the needs of the military. Thirty years of experience in Air Force
research projects, spanning from military programmer to Principal Investigator,
provides templates of reasonable courses of action.
Describe the difficulty or complexity and originality of your
work. Provide examples and
documentation.
Some information
furnished by incumbent during audit on 18 Aug 2004. Any additional documentation/information to
facilitate in classifying the position is appreciated.
The complexity and
originality in the work are essential elements as the incumbent is usually
called upon when PhD consultants and in-house programming staff have
encountered an en pass.
SCOPE AND EFFECT:
Describe the purpose of your work.
The focus of the
core job is to maintain the operational functioning of the Air Force Testing
Program in light of requests for policy exceptions and to validate major
selection/classification instruments used to select all Air Force Officers,
select/classify all enlisted personnel, and ensure a fair promotion system that
maintains the confidence of all airmen and the respect of civilian psychometric
experts. Extensions to the core job are required to ensure the success of the operational testing
mission in an environment of erosion of both support systems and other
expertise; and the ability to ensure the testing programs adapt to remain valid
in the context of the increasingly dynamic forces of a changing Air Force.
PERSONAL CONTACTS:
Describe your personal contacts.
Personal contacts include
advising and supporting AF/DPX on issues impacting on the Air Force Testing
Program – both operational concerns and forward-looking research needs and
technology advancement proposals. Other
personal contacts are with HQ agency focal points for reviewing proposed Air
Force assessment instruments (like the DO of HQ ROTC for the USAF Pilot Candidate
Interview Process, AETC SAS/CC concerning issues of AF pilot selection
automated Basic Attributes Test).
Additional personal contacts in the extended responsibilities include
participating in planning sessions (for example, Architecture Design of the
Enterprise Information Management –EIM System proposed for AFPC-wide
implementation) and interactions, coordination, advising, etc with outside
senior professional (PhD-level) personnel research and psychometrician
contractors, technical (Computer systems) personnel responsible for processes
impacting testing along with their senior AFPC personnel counterparts. Personal contacts are also maintained with
the professional psychometric community by representing the Air Force at
conferences such as the Annual Conference of the Military Testing Association
in
PURPOSE OF CONTACTS:
Using the contacts sited above, what was the purpose of your
contacts?
The purpose of contacts is
consultation and coordination in support of operational testing programs;
interactions related to research, development, and professional standards
concerning testing program issues; and achieve clarity on recommended
strategies for both operational accommodations and for long-term investment
decisions covering policy, procurements, and procedures. The purpose is achieved by working with
high-level professions, performing in-depth fact-finding, evaluating logistical
challenges and reaching a realistic level of agreement on proposed changes to
the status quo.
PHYSICAL DEMANDS:
Describe any physical
demands.
Similar to current job
description. Evidenced during audit on
18 Aug 2004.
WORK ENVIRONMENT:
Describe physical
surroundings in which you work.
Similar
to current job description. Evidenced during audit on
18 Aug 2004.