The development and performance validation of a tool to assess patient anticoagulation knowledge⋆

1. Introduction 

Suboptimal anticoagulation therapy is a significant problem because it may lead to hemorrhagic and thromboembolic events.1, 2, 3, 4, 5 This is primarily because of the complex and labor-intensive process involved with the clinical management of patients receiving warfarin anticoagulation therapy. There is substantial evidence to support that pharmacist-directed anticoagulation management of patients is effective and leads to improved health outcomes.6, 7, 8, 9, 10, 11 In addition, physician use of pharmacist-managed anticoagulation therapy and satisfaction with these services continue to grow.8, 12 The appropriate use of indication-based therapeutic ranges and standardized laboratory-reporting format (ie, International Normalized Ratio, INR) has helped health care providers better manage patient therapy. These tools provide information that is used by pharmacists to maintain optimal therapeutic effectiveness for patients receiving warfarin. However, while these tools are necessary, they alone are insufficient for achieving effective, successful, and sustained anticoagulation therapy. Comprehensive patient education must be included.2, 7

Achieving therapeutic effectiveness and optimal outcomes for patients receiving warfarin remains complex, and is often a challenge to pharmacists and other health care providers because of the influence of genomics, comorbidities, patient's age, additional prescription and/or over the counter medications, herbal products, dietary variability (eg, vitamin K intake, anorexia), medication compliance, and metabolic states (eg, fever, thyroid conditions).13 Because of the complexity of these variables, the patient's command of anticoagulation-related knowledge is an important component in maintaining optimal therapy and reducing adverse events. While pharmacists should and often do verify patient understanding verbally, these methods typically provide subjective evidence about what patients do and do not know. In addition, a verbal evaluation may overestimate the patient's comprehension of anticoagulation knowledge.

In a prospective study of Chinese patients receiving anticoagulation therapy and anticoagulation education, patient knowledge of their anticoagulation therapy (warfarin) and INR control was evaluated.12 Sixty-six patients were asked to answer 9 core warfarin knowledge questions. Overall, patient knowledge of anticoagulation therapy was poor, with the elderly and patients with lower literacy demonstrating the least understanding. This study gave evidence to support that the greater a patients' knowledge concerning his/her anticoagulation therapy, the better his/her INR control. A limitation, however, was that only 4 INRs were reviewed per patient.

Ideally, a written knowledge assessment would provide objective evidence that could be used to help pharmacists better understand what each patient knows about his/her therapy. The addition of patient-written knowledge assessments in diabetes education programs has demonstrated their usefulness to diabetes educators by identifying patient knowledge deficits and assisting educators in providing constructive feedback to patients and caregivers who contributed to health outcome improvement.14, 15 However, to date, no known written patient knowledge assessments have been reported in the anticoagulation arena. The purpose of this study was to develop the Anticoagulation Knowledge Assessment (AKA) instrument and provide reliability and validity evidence to support its use in the anticoagulation patient population. Ultimately, this assessment method will help pharmacists assess patient knowledge deficits. Thus, it will facilitate the provision of effective and efficient anticoagulation education that is targeted to each patient's specific needs. This study reports the results of the pilot testing of the 32-item AKA assessment instrument.

2. Methods 

A review of the published literature and multiple anticoagulation clinic protocols was performed to identify educational content areas typically addressed in pharmacist-managed anticoagulation clinics. Interviews with anticoagulation pharmacists were conducted to gain additional insight into their practices and they were asked to list educational content areas that they felt were essential for patients to understand. Additionally, the information provided by the anticoagulation pharmacists delineated common misconceptions or misunderstandings that patients have concerning their anticoagulation therapy. During the interview process, 10 major educational content areas were identified and these provided a template for the development of the assessment blueprint (Table 1).6, 9, 12, 16, 17, 18, 19, 20, 21

Table 1. Ten major educational content areas

	Content area	Examples
	Medication	Rationale for therapy
		How warfarin will benefit patient
		Planned length of therapy
		Dose
	Medication administration	When to take doses
		Storage of medication
		What to do in case of missed dose
	Medication interactions	New prescription medications
		OTC medications
		Herbal medications
	Activity	Fall precautions
		Contact sports
	Diet	Vitamin K and dietary considerations
		Alcohol use
		Dietary consistency
	Side effects	Signs of over anticoagulation
		Signs of disease recurrence
		What to do in case of bleeding
	Pregnancy	Contraindication in pregnancy
	Informing health care providers	Pharmacists
		Nurses
		All physicians
		Paramedics
		Dentists
		Coumadin Clinic
		Identification bracelets
	Procedures	Surgery
		Dental work
		Other procedures
	Laboratory monitoring	PT/INR
		Frequency of monitoring

OTC, over the counter.

A multiple-choice question format was chosen for the development of the AKA instrument because it is the most appropriate and efficient mechanism for assessing cognitive knowledge.22, 23 When information obtained from multiple-choice question items is analyzed using Rasch analysis, it provides data for the development of an instrument which will measure each patient's knowledge on an interval scale.24, 25, 26 These properties allowed the impact of the educational interventions provided by the pharmacist (eg, patient counseling, teaching) to be measured objectively and evaluated. Thirty-two items were drafted to represent the 10 educational content areas.

3. Results 

Study participation was voluntary. Sixty of the 100 patients asked, agreed to participate in the study and completed the AKA instrument (ie, 30 patients from the inner-city clinic and 30 patients from the suburban clinic). All patients were able to complete the instrument in approximately 20 minutes. Demographics for those who completed the instrument are shown in Table 2.

Table 2. Sample demographics

	Sample size	60
	Age in years	63 ± 19
	Race and ethnicity	(n)	%
	White	32	53.3
	African American	20	33.3
	Hispanic or Latino	7	11.7
	American or Alaska Indian	1	1.7
	Gender
	Male	21	36.4
	Female	39	63.6
		Mean	sd
	Number of clinic visits	59	42
	Number of months receiving warfarin	28	30
	Education in years	12	2.8

Most patients at the suburban clinic were white (one Alaskan or Native American) while the inner-city patients identified themselves as African American (25 patients), Hispanic (6 patients), or white (3 patients). The average age of participating patients was 63 years (SD ± 19). Twenty-five of the 30 patients from the suburban clinic were older than 65 years, however, at the inner-city clinic, only 10 of the 30 patients were older than 65 years.

One consideration of validity refers to the sampling adequacy of the content area being measured. Content validity for the AKA instrument was supported through the use of Marzano's Taxonomy, which is similar to Bloom's Cognitive Taxonomy.29 The taxonomy verified that the AKA instrument provided varying levels of cognitive difficulty ranging from simple recall of facts to application and use of knowledge. Additionally, to ensure that the multiple-choice question items of the AKA instrument were formatted well (ie, to minimize the error collected), each item underwent a quality control check using the 31-item multiple-choice question appropriateness checklist of Haladyna et al.27

An important consideration in the development of AKA items was that the instrument was designed to collect information from the general population including those with lower levels of literacy (ie, sixth-grade reading level). The final readability (grade level) of the pilot instrument was 7.6 using Microsoft Word XP readability statistics. The self-reported levels of education for the sample population varied greatly and are provided in Table 3.

Table 3. Highest level of education completed

	Grade	Frequency	Percent	Cumulative percent
	1	1	1.7	1.7
	8	2	3.3	5.0
	9	3	5.0	10.0
	10	2	3.3	13.3
	11	4	6.7	20.0
	12	29	48.3	68.3
	14	5	8.3	76.7
	15	1	1.7	78.3
	16	10	16.7	95.0
	18	3	5.0	100.0
	Total	60	100.0

Rasch analysis of the 32-item AKA pilot instrument identified 2 items exhibiting OUTFIT statistics greater than 1.2. These 2 items were related to pregnancy (see Appendix 2) and deleted from further analysis because they did not meet the unidimensional requirements of the Rasch model. In addition, the easiest item for subjects to answer correctly exhibited measurement redundancy (ie, MNSQ OUTFIT 0.29) demonstrating that responses for this item were over predicted. That is, while this information is important, it was generally well known to this population and contributed little to the measurement properties of the instrument (see location of item 30 in Figure 1). Evaluation of INFIT and OUTFIT statistics for the remaining 29 items in the AKA instrument showed that MNSQ values were less than 1.2 and greater than 0.8. Additionally, all items demonstrated point-biserial correlation values ranging from 0.20 to 0.53. Thus, these data exhibited good model fit and supported the unidimensionality and local independence requirements of the model. All 60 patients demonstrated person misfit statistics less than 1.3 indicating that the instrument functioned well in the sample population.

View Large Image Download to PowerPoint

Figure 1. Rasch item and person map.

The separation index (ie, the extent that items are sufficiently spread out to define distinct levels of measurement ability) for the 29-item AKA instrument was 2.8, which translates to an item reliability (ie, the estimate of reproducibility of item placement within the hierarchy of difficulty across patients of differing abilities) of 0.88. This indicates that the items created a variable that was well spread out on the measurement scale and that item placement along the scale was reliable. The separation index for the 60 patients was 1.8 that represents a person reliability coefficient of 0.75 (analogous to Cronbach's coefficient alpha).

An item distribution map was constructed to display visually the distribution of the patient ability measures and item difficulty calibration values on the same measurement scale in logits as can be seen in Figure 1. The scale measuring the AKA knowledge proficiency construct is laid out vertically with the most able persons and most difficult items at the top. This map shows visually the relationship between the patient's cognitive knowledge performance and item difficulty. The left-hand column locates the patient's ability and the right-hand column locates the item difficulty placement. The items are color coded to facilitate identification of the specific educational content areas that each of the items represented. Overall, the AKA instrument demonstrated an item difficulty distribution that was well targeted to the patient's population. That is, from a measurement perspective, each patient's ability measure (ie, as depicted by each “X” in Figure 1) was assessed adequately by items with calibration values in the same region of the measurement scale. Specifically, the educational content area related to medication information and side effects was well represented and distributed throughout the measurement continuum. Items representing diet educational content were located higher on the difficulty continuum than the other content areas as observed in the Rasch item/person map (see Figure 1).

4. Discussion 

The final version of the revised AKA instrument (Appendix 1) contains 29 items located at different levels of difficulty that allowed differences in patient knowledge about the anticoagulation care patients receive to be evaluated. Assessment items for the educational content areas “medication information/administration” and “side effects” demonstrated exceptionally good content coverage as demonstrated by the distribution of their placement throughout the item hierarchy.

This version of the AKA instrument was designed for use in a population where patients speak and/or write in the English language. Thus, patients who could not speak and/or write in the English language were excluded from the study. The authors are in the process of developing and equating a Spanish version of the AKA (ie, SAKA) to accommodate this population.

The patient demographics were different for the 2 data collection sites. However, the investigators believed that the mix of the demographics from the 2 practice sites provided a variation of experiences and knowledge to assist in the validity evaluation of the AKA.

The self-reported highest grade level of education completed by most patients was 12 years or more (n=48, 80%) with one subject self-reporting a grade level less than the eighth grade. However, the researchers noted that a few patients requested the AKA to be read to them. This may suggest that the level of literacy for some individuals was less than the self-reported level of completed education. This finding is consistent with those reported in 2 studies that evaluated the self-reported grade level of literacy of patients receiving anticoagulation (ie, 12th-grade education but who were reading at an 8th-grade level).39, 40 The AKA instrument was designed to be completed by patients with lower than average reading skills and demonstrated a reading level of 7.6 using Microsoft Word XP readability statistics. However, the authors believe that the AKA instrument functions at less than a seventh-grade reading level because many of the terms used in the instrument are those that patients become familiar with in the anticoagulation setting (ie, Coumadin, warfarin). Therefore, these clinically specific terms may artificially inflate the reading level as represented by the readability statistics. For example, when the words “Coumadin” and “warfarin” were removed from the instrument, the readability statistics demonstrated a reading level of 5.5. Regardless of whether the instrument was completed by actually reading the items or having the items read to them, the person fit statistics supported that the content of the items as represented in the AKA instrument was understood. That is, all 60 patients demonstrated person misfit statistics less than 1.3 indicating that the instrument performed well in the population in which it was administered and that the items were understood by each individual.

Constructing items for an assessment instrument is a challenging task that requires careful consideration.41 The Rasch model assisted the researchers by providing information that served as quality control for item functioning. The 2 educational items concerning pregnancy that exhibited unacceptable MNSQ OUTFIT statistics were deleted because they did not meet Rasch model requirements. In essence, these items did not function unidimensionally with the other assessment items. The authors believe this was because these 2 items were less applicable for men and women older than 55 years (ie, beyond childbearing age). Because women of childbearing age would be less likely to be completing the AKA instrument (ie, less likely to be receiving warfarin therapy), items relating to pregnancy were not included in the final version of the AKA. However, when a woman of childbearing age is receiving anticoagulation, the authors recommend that these patients be asked information about pregnancy as relates to warfarin to obtain a more comprehensive evaluation. These two items are included in Appendix 2.

It is interesting to note that items concerning diet were more difficult for patients to answer correctly. This finding supports that this content area represented a more challenging area for patients to understand and is consistent with the findings of a written knowledge assessment for use in patients with diabetes.15

Rasch model analysis helped the investigators determine how the AKA can be improved. A small gap in the measurement scale was identified at the upper end of the difficulty range (as can be seen in Figure 1); the addition of items targeted to these difficulty levels may increase the AKA's ability to detect change in cognitive knowledge exhibited among patients of higher ability over time. However, the addition of items in these areas may not be necessary if future studies support that patients who attain higher ability levels demonstrate stable and therapeutic INR values. That is, the presence of these small measurement gaps may be of little clinical importance.

The validity evidence provided by analyses of the data collected in this study supported that AKA instrument performed well in the population in which it was administered. The AKA instrument demonstrated good item and person reliability indices supporting that the items gather information in the anticoagulation population in a manner that will reproduce similar results when re-administered in future applications. Additionally, the AKA instrument demonstrated that items calibration values were well targeted to the population and that no ceiling or floor effects were present. This allowed the detection of differences in individual patients. This measurement property of the instrument is an important consideration in detecting and tracking changes in the patient's knowledge over time.

The AKA instrument was designed to be used primarily as a continuous quality improvement tool for use by clinical pharmacists who direct anticoagulation programs. It is expected that when knowledge deficits in patients are identified with the administration of this instrument early in therapy, an educational intervention will be developed that will lead to better patient understanding of their anticoagulation therapy. Therefore, it makes sense that the patients receiving this education would more quickly achieve and maintain prothrombin time (PT)/INR stability, thereby reducing the incidence of adverse events. Likewise, patient results reported in aggregate by each clinic using the AKA instrument may be used to improve the anticoagulation programs' effectiveness in the delivery of patient education.

The piloting of the AKA instrument has already demonstrated its usefulness as a continuous quality improvement tool. Clinical pharmacists participating in this pilot study reported to investigators that several patients had specific questions arising directly from their completion of the AKA instrument. These questions related to clarification of specific educational content areas and provided patient-driven opportunities for the pharmacist to further develop a pharmacist/patient relationship. This was an unexpected but well-received finding. Initially, there were concerns that administration of the AKA instrument would pose a significant burden to the patients; however, patients were stimulated to engage the pharmacist, thereby, facilitating patient/pharmacist interaction and re-education of unclear or misunderstood educational content areas.

The AKA instrument can provide good continuous quality improvement to measure performance of program facilitators. Providers should expect that a patient understands certain concepts after a specified number of visits. Ultimately, the goal is that the patient incorporates the information gained from his/her encounter with the clinician into daily use, empowering patients to take charge of their own health care. Use of the AKA instrument will identify concepts that the patient has a less than complete understanding of. Identification of these gaps in knowledge will allow the pharmacist to focus on each of these content areas during follow-up clinic visits.

The authors suspect there is a relationship between the level of patient knowledge about anticoagulation and the attainment of therapeutic PT/INR, stability over time, and the incidence of adverse events. Thus, this relationship will be investigated during the next phase of the AKA validity studies.

5. Conclusions 

These data clearly show the practical application of validation techniques and Rasch analysis in designing and evaluating information obtained from the patient population that is meaningful in its interpretation. The information gained from using the AKA instrument will provide pharmacists with direction for anticoagulation management education that is targeted to each patient's specific needs. Additionally, the patient's responses will provide pharmacists with objective data about those components of practice that are being taught effectively. Thus, the instrument also serves as quality control for pharmacist/patient counseling effectiveness. The methodology used in the development of AKA instrument provides a template that can be used to produce assessment instruments for other clinical areas (eg, diabetes, asthma, dyslipidemia).