With constructs defined and content validity established, the next task is writing the actual measurement items. This is where most controllable measurement error gets introduced.
Flaws built into items do not disappear in analysis. Double negatives that respondents misread. Double-barreled questions that force a single response across two distinct judgments. Leading language that nudges respondents toward a particular answer. These problems inflate error variance and, in some cases, systematically bias observed relationships between variables. Reviewers who raise common method bias concerns are pointing to a problem that was created at the design stage — not the analysis stage.
The item design stage combines three tasks in a single workflow: generating items from factor definitions, detecting and removing linguistic flaws, and controlling for response bias structurally.
Writing Items from Factor Definitions
Each item should be written from the operational definition of the factor it is intended to measure. The definition specifies the construct boundary — what counts as evidence of the construct and what does not. Items that fall outside that boundary are not measuring the intended construct, regardless of how they perform statistically.
A Likert-scale item has several components that each require a decision: the stem (the declarative statement), the response scale (typically 5 or 7 points), the anchors (verbal labels at each end), and whether the item is positively or negatively keyed. The stem should address a single judgment, be phrased at the reading level of the target respondents, and refer to a specific and consistent time frame.
A common recommendation is to generate a pool of candidate items — typically two to three times the number intended for the final scale — and then reduce through expert review and pilot analysis. This redundancy allows poorly performing items to be removed without losing construct coverage.
Detecting Linguistic Flaws Before Data Collection
Double Negatives
A double negative combines two negation markers in a single item: "I do not feel that my supervisor fails to support me." The respondent must reverse the logic twice to understand what direction of agreement is being asked for. This produces interpretation errors even among respondents with strong reading comprehension.
Reverse-keyed items do not require double negatives. The correct way to reverse an item is to reverse its direction simply and directly. "I am satisfied with my work" becomes "I am not satisfied with my work" — not "I do not find my work unsatisfying."
Double-Barreled Questions
A double-barreled item asks about two distinct things simultaneously: "My supervisor's feedback is fair and constructive." Fairness and constructiveness are separate judgments. A respondent who considers the feedback fair but not constructive has no coherent answer to give. The item should be split into two or simplified to one dimension.
Double-barreled items are identified by looking for conjunctions — and, but, or — connecting two evaluative claims.
Leading Language
A leading item steers the respondent toward a particular answer through its phrasing: "Our talented team leader listens carefully to team members' ideas." The embedded judgment ("talented") presupposes a positive evaluation before the question is even asked. The item captures confirmation, not measurement.
Linguistic Flaw Reference Table
| Flaw type | Example | Detection criterion |
|---|---|---|
| Double negative | "does not fail to support" | Two or more negation markers in one item |
| Double-barreled | "fair and constructive" | Two distinct evaluative claims joined by a conjunction |
| Leading language | "our talented leader" | Embedded judgment or assumed premise |
| Ambiguous time frame | "I usually feel…" | No specific or consistent temporal reference |
| Jargon overload | "I engage in organizational citizenship behaviors" | Terminology respondents may not recognize |
Reverse Items and Common Method Bias Prevention
Removing linguistic flaws is necessary but not sufficient. Response bias can still contaminate data through a different mechanism. Common Method Bias (CMB) is systematic measurement error that arises when independent and dependent variables are collected from the same respondent using the same method at the same time. CMB is not fixable in analysis — it must be controlled at the design stage.
Reverse-Keyed Items
Reverse-keyed items are the most basic structural control against acquiescence bias — the tendency of respondents to agree with items regardless of content. When all items point in the same direction, a respondent can produce a consistent-looking pattern of high scores without reading individual items. Reverse items disrupt this.
A commonly applied range is 20–30% of total items. There is no single universally mandated ratio in the measurement literature, but this range reflects the balance between disrupting acquiescence and avoiding the cognitive confusion that comes from too many reversed items. Reverse items should be distributed throughout the scale; once respondents notice the pattern, the disruption effect is lost. When writing them, reverse the direction simply — not through a double negative.
Marker Variables
A marker variable is an item or set of items theoretically unrelated to the research model but susceptible to the same response biases as the study variables. Including one allows the researcher to estimate the magnitude of common method variance after data collection, using the procedure described by Lindell and Whitney (2001). The observed correlations between the marker and the study variables provide a lower-bound estimate of CMB contamination, which can be partialled out as a statistical adjustment.
The marker variable must be genuinely unrelated to the substantive constructs. If the wrong variable is chosen, the adjustment is meaningless. Researchers using this technique for the first time should consult the original Lindell and Whitney (2001) procedure before applying it.
Temporal and Methodological Separation
The cleanest design-level control is to separate the measurement of predictors and outcomes across time or method. A two-wave design substantially reduces the co-occurrence conditions that produce CMB. Mixing self-report with observer ratings or archival data achieves a similar effect. A single-source, single-time, single-method design is the most vulnerable. If the research question permits an alternative design, that choice should be made before item writing begins.
Post-Hoc Verification
Design controls reduce CMB; they do not eliminate it. After data collection, Harman's single-factor test is the standard minimum check: if a single factor explains less than 50% of total variance in an exploratory factor analysis (Harman, 1976), CMB is not considered an overwhelming concern. Where a marker variable was included, the Lindell and Whitney (2001) procedure provides a more precise estimate and statistical adjustment.
modidoc and the Item Design Stage
modidoc's item design stage generates a Likert-scale item pool from factor definitions, then automatically flags double negatives, double-barreled constructions, and leading language with suggested revisions. Reverse item inclusion and distribution are managed within the same workflow. This stage is implemented internally as the C2 item generation engine.
Frequently Asked Questions
What proportion of items should be reverse-keyed?
A practical range of 20–30% is widely applied in scale development. No single universally mandated ratio exists in the measurement literature. Distribution within the scale matters as much as proportion — reverse items should be spread throughout so that respondents cannot identify and adapt to the pattern.
How do you prevent common method bias in survey research?
At the design stage: include reverse-keyed items, add a marker variable unrelated to the theoretical model, and consider temporal or methodological separation between predictor and outcome measurement. After data collection, Harman's single-factor test (single factor < 50% of variance) is the standard minimum check. The Lindell and Whitney (2001) marker variable procedure provides a more precise estimate where a marker was included.
Why are double negatives a problem?
A double negative requires respondents to mentally reverse two statements to determine which direction of agreement is being asked for. This produces interpretation errors that add noise to the measurement rather than capturing the intended construct. Reverse items should state the reverse direction once, directly and simply.
Next
With item design complete, the next step is verifying that respondents interpret the items the way the researcher intended. Misinterpretation is more common than response distributions reveal, and it cannot be detected without directly asking respondents to think aloud. The next article covers the respondent verification stage: Think-Aloud cognitive interviewing and the protocol for identifying interpretation mismatches before full data collection.
Previous: Construct Extraction and Factor Structuring — A Complete Guide to Scale Development Step 1