Validation Study of the Postoperative Cognitive Dysfunction Database in Siriraj Hospital, Thailand

Surapa Tornsatitkul, M.D.1, Patumporn Suraarunsumrit, M.D.2 , Laddawan Jensarikit, M.Sc.3, Arunotai Siriussawakul, M.D., Ph.D.1,4 , Suvimol Niyomnaitham, M.D., Ph.D.3,5,*

1Department of Anesthesiology, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok 10700, Thailand, 2Department of Medicine, Faculty

of Medicine Siriraj Hospital, Mahidol University, Bangkok, 10700, Thailand, 3Department of Pharmacology, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok 10700, Thailand, 4Integrated Perioperative Geriatric Excellent Research Center, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok 10700, Thailand, 5Siriraj Institute of Clinical Research, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok 10700, Thailand.

ABSTRACT

Objective: Postoperative Cognitive Dysfunction (POCD) is a complication that arises in the elderly. Because of the limited knowledge of POCD, researchers must handle a substantial amount of data to ensure the comprehensive collection of all relevant factors. To deal with this data, a validation study is a valuable method that aids in qualifying the data.

Materials and Methods: A validation exercise was performed for 40% of the data in the Siriraj POCD database (n=250) in 2020-2023. The validation covered 30 items, including demographic data, surgical and anesthetic factors. The validation study had two components: internal validation, which aimed to assess the completeness, uniformity, plausibility, and accuracy of the data in the database, and external validation, where the results were compared to external literature to confirm their correspondence.

Results: The completeness was 99.2% for creatinine and 94.0% for hemoglobin, while others showed 100% completeness. The accuracy ranged from 73.6% to 99.6%, with a median of 97.4%. Most errors found were related to “body weight”, followed by “hemoglobin levels” and “Propofol targeted controlled infusion”, with accuracy rates of 73.6%, 84.0%, and 85.2%, respectively. In the external validation, the POCD incidence at 1 week from surgery in the literature review ranged from 8.9%–46.1% compared to 26.0% in our study.

Conclusion: The Siriraj POCD cohort study database was found to be reasonably valid. Therefore, this data can support high-quality research. Our recommendations for developing a good database include implementing a dedicated plan, employing trained staff, and using reliable data sources.

Keywords: Postoperative cognitive dysfunction; validation study; validate; database; Risk factors (Siriraj Med J 2024; 76: 429-435)

INTRODUCTION

Postoperative Cognitive Dysfunction (POCD) is one of the postoperative consequences in the elderly leading to a higher mortality rate, longer length of hospital stay, independency, and a poor quality of life.1-3 The International Society of Postoperative Cognitive Dysfunction defines POCD as a decline in cognitive function in more than

1 domain after surgery. Currently, there is no definitive treatment for POCD, so prevention is the primary focus. The pathogenesis of POCD remains unclear, while the incidence of POCD varies from 8.9%–46.1% depending on the patient characteristics, surgical type, and diagnostic criteria.4,5 Old age, lower educational level, and previous stroke are proven to be risk factors.6 Nevertheless, there

*Corresponding author: Suvimol Niyomnaitham E-mail: suvimol.niy@mahidol.edu

Received 29 February 2024 Revised 20 March 2024 Accepted 21 March 2024 ORCID ID:http://orcid.org/0000-0002-4785-2716 https://doi.org/10.33192/smj.v76i7.268007

All material is licensed under terms of the Creative Commons Attribution 4.0 International (CC-BY-NC-ND 4.0) license unless otherwise stated.

is limited understanding of other associated factors, particularly those related to anesthesia and surgery. High- quality research is thus needed to enhance knowledge in this field.

Recent evidence suggests that POCD may be associated with various factors during the preoperative, intraoperative, and postoperative periods. Multiple medical fields, including anesthesiology, surgery, psychiatry, and internal medicine, are involved in research in this area. At our center, Siriraj Hospital, Mahidol University, Thailand, data collection primarily relies on written medical charts rather than electronic computerized records. This poses a significant challenge, and so a dedicated research team, known as the Siriraj POCD cohort study team, was established to gather essential data related to POCD.

As mentioned earlier, the data related to POCD is extensive and diverse, and there is a risk of inaccurate data, which could cause misleading results for research and further treatment. Therefore, it is crucial to validate the data to ensure its reliability which is a vital aspect of conducting high-quality research and improving patient care. This process is referred to as database validation. A practical approach framework was developed by Hoeven et al.7, which includes internal validation and external validation. Internal validation assesses the validity of data within a single data source, considering factors like accuracy, completeness, uniformity, and plausibility. Here, accuracy checks whether the data is correct when compared to reliable sources, completeness ensures that no variables are missing, uniformity ensures that data is recorded consistently in the same units and coding system, and plausibility verifies that the data seems reasonable. In external validation, data is compared to that in external sources, such as other literature and expert opinions, to ensure consistency. While the concept of database validation is straightforward, it is not always widely practiced or reported as objective data. Our objective was to validate the Siriraj POCD cohort database in terms of its completeness, accuracy, uniformity, plausibility, and concordance with external sources.

MATERIALS AND METHODS

This cross-sectional study received approval from Siriraj Institutional Review Board 623/2563(IRB3) at August 19, 2020. The data was retrieved from POCD database in 2020-2023. According to the sample size estimation for this study, there was no consensus regarding the adequate sample size for a validation study. Herrett et al.8 demonstrated that the sample size for the manual review of computerized records ranged from 33% to 100%, with a median of 86.2%, based on data from 31 studies.

Therefore, a sample of 250 subjects was selected from the Siriraj POCD cohort database, which represented 40% of all the available data.

The Siriraj POCD cohort study database includes patients aged ≥ 65 years old who have undergone major cardiac or non-cardiac surgery. The recruitment process for the POCD database started in November 2017 and continued until 2023, with a total sample size of 625. The main outcomes in the Siriraj POCD cohort are the incidence of POCD and associated factors. POCD is diagnosed at our center by a drop of two or more points in the Montreal Cognitive Assessment (MoCA) score from the preoperative score compared to the postoperative score at 1 week after surgery. The data was independently reviewed by two individual reviewers (anesthesiologist and anesthesiology resident) directly from the primary source and documented in REDcap. If discrepancies arose, the reviewers conducted a reevaluation to ensure accuracy. The data from primary sources were recorded in REDcap and exported to csv file (Data 1). The data from Siriraj POCD database were also exported to csv file (Data2). The patient ID form 2 dataset was matched to ensure the same ID was compared. The accuracy was analyzed by RStudio ‘arsenal’ orderset and reported as percentage accuracy. The completeness, uniformity, and plausibility were analyzed by RStudio and reported as a percentage. POCD incidence and odds ratios of associated factors were analyzed by IBM SPSS Statistics (version 29; IBM 126 Corp, Armonk, NY, USA). P-value < 0.05 was considered statistically significant. The results were compared to other literature. The validation workflow was depicted in Fig 1.

Data items

The following 30 items as potential associating factors of POCD from previous literature were chosen to validate the data.

I: Preoperative data: patient identification number, operative date, age, gender, weight, height, diabetes mellitus, hypertension, diazepam, lorazepam, serum creatinine, serum sodium, and hemoglobin

II: Intraoperative data: American Society of Anesthesiologist Physical Status Classification (ASA), site of surgery, anesthetic technique, start operative time, finish operative time, blood transfusion, inhalation, induction agent, analgesic drug, midazolam, dexmedetomidine, inotropic drug, bispectral index monitoring, and near- infrared spectroscopy (NIRS) monitoring

III: Postoperative data: POCD incidence, length of stay, in-hospital mortality.

Fig 1. Study flow

Internal validation7

Completeness was assessed by ensuring there were no missing variables. Leaving a blank was not acceptable, and it could not be interpreted as a “No.” If the required data could not be obtained from the patient, it should be reported as “not applicable (N/A)”. If the data indicated a negative response (i.e., no use of a drug or method), it should be recorded as a “No”. This approach guaranteed a comprehensive review of all the data, covering both the positive and negative aspects.

Uniformity was evaluated by examining the diversity of recording systems and reporting the findings as percentages. The operative date was expressed as a percentage recorded in the AD dating system. Weight and hemoglobin were reported as percentages with one decimal place, and serum creatinine with two decimal places. Variations in diagnoses were also carefully observed and documented. Plausibility was examined by ensuring that the data conformed to expected and acceptable numerical value ranges. In this study, certain parameters were expected to fall within the following historical ranges: 130–190 cm for height, 5–18 g/dL for hemoglobin, 0.3–5 mg/dL for serum creatinine, and 110–150 mmol/L for serum

sodium.

Accuracy was verified by cross-referencing the data with the primary source. The primary source was considered the initial and most reliable place to obtain the data. While some data could be found in various forms within written medical records, the primary data needed to be collected from the most dependable and up-to-date source available. For instance, blood test results were obtained from the hospital’s laboratory report program, and medication information was sourced from the anesthetic records.

External validation

Following the internal validation, any inaccurate data item found was corrected to enhance the quality

of the Siriraj POCD database. The preliminary results, including the incidence of POCD and the odds ratios of the risk factors, were then compared to findings from other literature sources.

RESULTS

A total of 30 variables of 250 patients were checked for completeness. Two data points of serum creatinine and 15 data points of hemoglobin were missing. This corresponds to 99.6% completeness for serum creatinine and 94.0% completeness for hemoglobin. The remaining items showed 100% completeness. For uniformity, the operative year was recorded in the AD dating format for 96.4% of cases, while the remaining operative dates were incorrectly recorded in the Buddhist Era (BE) dating format. Weight was mostly recorded with one decimal place in 60.8% of cases, and hemoglobin was reported with one decimal place in 82% of cases. Serum creatinine values were recorded with two decimal places in 85.1% of cases. Diagnoses exhibited significant diversity in terms of abbreviation, specificity, and word order. For plausibility, height, hemoglobin, serum creatinine, and serum sodium were 100%, 93.2% 97.6%, and 91.2% within acceptable ranges, respectively.

The accuracy range was 73.6%–99.6% with a median of 97.4%. The accuracy results are shown in Table 1. Out of 41 items, 25 items were found to have an accuracy rate of more than 96%. The most error-prone factor was “weight,” with an accuracy of 73.6%. Following that, “hemoglobin” had an accuracy of 84%, and “propofol continuous infusion” had an accuracy of 85.2%.

In terms of external validation, the POCD incidence at 1 week from surgery from the literature review ranged from 8.9%–46.1%, while our result showed an incidence of 26.0%. Additionally, the odds ratios of the associated factors were compared to those in the external literature (as shown in Table 2).

TABLE 1. Accuracy of the data items checked (n = 250)

Abbreviations: ASA = American Society of Anesthesiologist Physical Status Classification, Cox2 inhibitor = Cyclooxygenase-2 inhibitor, NSAIDS = Non-steroidal anti-inflammatory drugs

DISCUSSION

Our validation study demonstrated robust internal validation, with completeness nearing 100%. Uniformity and plausibility were within an acceptable range, and accuracy ranged from 77.6% to 99.6%, with a median of 97.2%.

There is no definite cutoff point indicating high- quality data per se. However, when it came to completeness, data were considered excellent if approaching 100% completeness. In the Siriraj POCD database, completeness was 100% for most data items; however, two items, serum creatinine and hemoglobin, fell slightly short, with completeness rates of 99.2% and 94.0%, respectively. The main reason for missing these laboratory values was the absence of a blood test. Leaving the record blank, which usually indicates uncollected data, can create ambiguity. Therefore, to improve the quality of data in term of completeness, we encourage researchers to define a variable for negative results, such as recording with a ‘No’ or ‘0’ and a separate variable for a result that is not applicable in some data cells, such as ‘NA’.

For uniformity, some data were recorded in different units or with varying decimal places. To correct these inconsistencies, it is recommended to use a well-structured dedicated case record form. Each variable should be specified in detail. For numerical values, the form should indicate the number of decimal places to be used. In the case of diagnosis and procedures, adopting internationally recognized and reliable coding systems, such as the International Classification of Diseases (ICD), is advised to ensure uniformity and consistency in data recording. To ensure plausibility, data that fell outside the acceptable range were rechecked. Two data points for serum creatinine were incorrect, mistakenly recorded as BUN (Blood Urea Nitrogen), while the remaining serum creatinine data were accurate. Additionally, two data points for hemoglobin were erroneously corrected from hematocrit, and four serum sodium values were inaccurately corrected from BUN. The remaining values were recorded as ‘0.0’. A computerized case record form with range restrictions for input could have helped

prevent and correct these errors.

In terms of accuracy, previous studies have shown a wide range, with accuracy varying from 45% to 100%. In our study, the median accuracy was relatively high at 97.4%. Weight was the most inaccurate data point in our study, with an accuracy of 73.6%. The primary reasons for this inaccuracy were that weight can change day by day and was often rounded up from one decimal place to an integer value. Other factors in the top five inaccurate items, including weight, hemoglobin, serum creatinine,

TABLE 2. External validation of the odds ratios with external literature

Variables

Siriraj POCD database Univariate analysis

Crude OR (95%CI)

p-value

Other literature

OR (95%CI)

p-value

Patient factors

Age (years)

Age < 70 years old 1.58 (0.823.04) 0.167 2.78 (1.13–6.85)1 0.025

Age ≥ 70 years old Ref

Education level

Abbreviations: OR = odd ratios, Ref = reference, MOCA = Montreal Cognitive Assessment

height and propofol targeted controlled infusion, which involve numerical data, and these might face similar issues related to value fluctuations and estimations by the data collectors. Theoretically, the value that is updated most closely before surgery should be chosen to improve accuracy. To enhance the quality of numerical data, it

is advisable to record data with specific decimal places and include the date and time details. Categorical data can be more complex. Errors may have arisen from clinicians or researchers, as many medical records are handwritten and may use detailed forms. Additionally, drugs and other medical details may be recorded using

full names, trade names, or abbreviations. To enhance accuracy, we recommend utilizing trained anesthesiology staff familiar with the anesthetic form for data collection.

In the context of external validation, the incidence of POCD in our study closely aligned with findings from external literature, suggesting good concordance. In detail, the odds ratios for postoperative delirium, midazolam use, blood transfusion, and surgeries lasting over 4 hours were consistent with those observed in external literature. However, in contrast, preoperative cognitive impairment, dexmedetomidine use, and NIRS monitoring exhibited opposite relationships compared to in the external literature. The main reason for the results showing the opposite trend in some cases can be attributed to the relatively small sample size and the statistical method that was employed. Also, univariate analysis was used, which did not effectively account for confounding factors. Other factors in the study did not show significant enough differences to warrant comparison.

CONCLUSION

The Siriraj POCD cohort study database is a high- quality database, but there is room for improvement, as with any database. Our recommendations for developing and maintaining an accurate database include the use of a well-structured computerized case record form, ensuring data is obtained from reliable sources with specific time points, and employing trained and experienced staff for data collection. Additionally, conducting frequent internal validation by a research team is advisable to ensure the database’s continued accuracy and quality.

Limitation

In the investigated hospital, certain data, like anesthetic records, are handwritten and recorded by trainees. This practice can introduce errors, as illegible handwriting can lead to misinterpretation of the data.

ACKNOWLEDGEMENTS

This research project was supported by National Research Council of Thailand (NRCT) (Grant Number R016541053). The funder had no role in the study. We are grateful to Rinrada Preedachitkul for her statistical support, as well as Arporn Pimtong for administrative support.

Conflicts of Interest

The authors have no conflicts of interest to declare.

Author Contributions

ST, AS, and SN were responsible for the

conceptualization. ST, AS, and SN handled the methodology. ST, AS, and SN oversaw validation. LJ carried out the formal analysis. AS managed resources. ST evaluated the data. ST drafted the original manuscript. ST, PS, LJ, AS and SN reviewed and edited the manuscript. AS supervised the project, handled administration, and acquired funding.

REFERENCES

1. Norkiene I, Samalavičius R, Misiuriene I, Paulauskiene K, Budrys V, Ivaškevičius J. Incidence and risk factors for early postoperative cognitive decline after coronary artery bypass grafting. Medicina. 2010;46(7):460-4.

2. Canet J, Raeder J, Rasmussen LS, Enlund M, Kuipers HM, Hanning CD, et al. Cognitive dysfunction after minor surgery in the elderly. Acta Anaesthesiol Scand. 2003;47(10):1204-10.

3. Suenghataiphorn T, Songwisit S, Tornsatitkul S, Somnuke P. An Overview on Postoperative Cognitive Dysfunction; Pathophysiology, Risk Factors, Prevention and Treatment. Siriraj Med J. 2022; 74(10):705-13.

4. Wang W, Feng N, Zhao W, Luo F, Zhu X, Zhao W, et al. Dexmedetomidine reduces brain neuronal injuries but not clinical neurocognitive function in the elderly, compared to midazolam. Int J Clin Exp Med. 2019;12(4):4210-7.

5. Steinmetz J, Rasmussen LS. Peri-operative cognitive dysfunction and protection. Anaesthesia. 2016;71 Suppl 1:58-63.

6. Brodier EA, Cibelli M. Postoperative cognitive dysfunction in clinical practice. BJA Educ. 2021;21(2):75-82.

7. Hoeven LRV, Bruijne MC, Kemper PF, Koopman MMW, Rondeel JMM, Leyte A, et al. Validation of multisource electronic health record data: an application to blood transfusion data. BMC Med Inform Decis Mak. 2017;17(1):107.

8. Herrett E, Thomas SL, Schoonen WM, Smeeth L, Hall AJ. Validation and validity of diagnoses in the General Practice Research Database: a systematic review. Br J Clin Pharmacol. 2010;69(1):4-14.

9. Silbert B, Evered L, Scott DA, McMahon S, Choong P, Ames D, et al. Preexisting Cognitive Impairment Is Associated with Postoperative Cognitive Dysfunction after Hip Joint Replacement Surgery. Anesthesiology. 2015;122(6):1224-34.

10. Feinkohl I, Winterer G, Pischon T. Diabetes is associated with risk of postoperative cognitive dysfunction: A meta- analysis. Diabetes Metab Res Rev. 2017;33(5).

11. Feinkohl I, Winterer G, Pischon T. Hypertension and Risk of Post-Operative Cognitive Dysfunction (POCD): A Systematic Review and Meta-Analysis. Clin Pract Epidemiol Ment Health. 2017;13:27-42.

12. Goldberg TE, Chen C, Wang Y, Jung E, Swanson A, Ing C, et al. Association of delirium with long-term cognitive decline: A meta-analysis. JAMA Neurology. 2020;77(11):1373-81.

13. Yang W, Kong LS, Zhu XX, Wang RX, Liu Y, Chen LR. Effect of dexmedetomidine on postoperative cognitive dysfunction and inflammation in patients after general anaesthesia: A PRISMA-compliant systematic review and meta-analysis. Medicine (United States)d. 2019;98(18).

14. Mansouri N, Nasrollahi K, Shetabi H. Prevention of Cognitive Dysfunctionafter Cataract Surgerywith Intravenous Administration of Midazolam and Dexmedetomidine in Elderly Patients

    Undergoing Cataract Surgery. Adv Biomed Res. 2019;8(1):6.

15. Punjasawadwong Y, Chau-In W, Laopaiboon M, Punjasawadwong S, Pin-On. Processed electroencephalogram and evoked potential techniques for amelioration of postoperative delirium and cognitive dysfunction following non-cardiac and non- neurosurgical procedures in adults. Cochrane Database Syst Rev. 2018;5(5):CD011283.

16. Ortega-Loubon C, Herrera-Gómez F, Bernuy-Guevara C, Jorge-

    Monjas P, Ochoa-Sangrador C, Bustamante-Munguira J, et al. Near-infrared spectroscopy monitoring in cardiac and noncardiac surgery: Pairwise and network meta-analyses. J Clin Med. 2019;8(12):2208.

17. Zhu SH, Ji MH, Gao DP, Li WY, Yang JJ. Association between perioperative blood transfusion and early postoperative cognitive dysfunction in aged patients following total hip replacement surgery. Ups J Med Sci. 2014;119(3):262-7.