Automating hybrid collective intelligence in open-ended medical diagnostics

Our approach uses a large dataset on general medical diagnostics collected by the Human Diagnosis Project (Human Dx). Human Dx is an online collaborative effort created to provide a global teaching environment for clinicians and to tap into the wisdom of the global medical community. It comprises an online platform to which medical experts can submit and solve patient cases. Patient cases consist of general patient information (e.g., age, gender, general symptoms) and a series of clinical findings, such as the outcomes of physical and diagnostic tests (e.g., laboratory and imaging studies; Fig. 1A). The medical experts creating the case know the correct diagnosis from further follow-up research. An expert panel reviews each case and decides whether the case is of sufficient quality and representative of the given domain. If so, the case is published and becomes accessible to the users of the platform. Cases may be removed from the platform if many users indicate that a case is problematic in terms of clarity or quality. Human Dx tags cases with a label indicating the prime specialty of a case, with cases stemming from a wide range of medical specialties (e.g., cardiology, dermatology, endocrinology, neurology, etc.).

Users from all over the world are invited to register at the platform and diagnose the uploaded medical cases. A case starts with showing general patient information and the first clinical findings (Fig. 1A). A user can proceed to the next finding of a case by button click. After observing all findings, they are asked to submit their diagnosis. The user can provide a single diagnosis or a ranked list of multiple diagnoses. Moreover, they can enter each diagnosis as free text or select an option from a catalog of medical terms (suggested as they type). After submitting their diagnosis, the user receives the correct solution of the case.

We analyzed all cases created between May 7, 2014, through October 5, 2016, for which at least 10 users provided a diagnosis (1,572 cases). For each case, 10 diagnosticians were randomly sampled from all diagnosticians who completed at least one other case before. In this way, we select only diagnosticians with some experience in using the platform. The sample included 2,069 unique users from 47 different countries, though predominantly from the United States (91%). We used self-reported tenure to determine the seniority of users (medical student, intern, resident, fellow, attending physician).

Arguably, the biggest challenge for the aggregation of independent diagnoses in open-ended medical diagnostics is to identify which diagnoses point to the same medical concept. This includes mundane challenges (e.g., British versus American spelling, capitalized letters or not, punctuation, typos, etc.) but also the thorny problem of determining whether or not two reported diagnoses are equivalent. To address these issues, we developed an automatic, reproducible, and scalable method to identify exact medical concepts from free-text diagnoses, which relies on a combination of semantic knowledge graphs and natural language processing (NLP) and integrates a publicly available medical ontology, i.e., the SNOMED Clinical Terms ontology (SNOMED CT). SNOMED CT is a systematically organized computer-processable collection of medical terms and considered the most comprehensive, multilingual clinical healthcare terminology in the world (35, 36) containing over 78,000 unique diseases (37). Specifically, our method leverages a semantic knowledge graph that we constructed by applying knowledge engineering techniques in order to reuse design best practices, e.g., ontology design patterns; (38) and linking information about medical cases and users’ diagnoses to SNOMED CT, which we exploit by gathering definitions for and taxonomic relations among clinical terms (Material and Methods and SI Appendix).

Fig. 1B illustrates our knowledge engineering approach. The knowledge graph is constructed with data from the Human Dx dataset, using knowledge-engineering techniques to extract the knowledge available from the dataset in terms of relations among concepts and subject–predicate–object triples (SI Appendix), matching the concepts related to clinical terms against the SNOMED CT ontology, which is therefore aligned within our knowledge graph. This is a two-step process. The first step is string normalization, whereby we use routine NLP tools to standardize all diagnoses (both the ones given by the users and the correct ones provided by a case’s author). This step consists of a series of text normalization procedures, including removing stop words, converting British English to US English, converting plural to singular, and identification of acronyms (see Material and Methods and SI Appendix for a complete description of the procedure). In the second step (concept mapping), we mapped each of these normalized diagnoses to an existing medical concept within SNOMED CT (July 2020 International Edition Release). For each diagnosis, we identified which SNOMED CT identifier(s) (SCTID) exactly matched a normalized diagnosis. To illustrate, the character strings “obstructive sleep apnea,” “osa,” “OSA,” and “OSA—obstructive sleep apnea” are all considered synonyms pointing to the same SCTID $78275009$ (SI Appendix, Fig. S1). The character string “sleep apnea,” however, points to a different SCTID (73430006; see also Fig. 1B). We considered only exact word matches (i.e., a Jaccard similarity of 1, see Materials and Methods; in SI Appendix, we show additional analyses relaxing this matching criterion). Occasionally, a character string showed an exact match with more than one SCTID (4.4% for correct diagnoses and 5.0% for users’ diagnoses). For example, the character string “Kaposi’s sarcoma” returned two SCTIDs named “Kaposi’s sarcoma (disorder)” and “Kaposi’s sarcoma, morphology (morphologic abnormality),” respectively. In such cases, we relied on the semantic tags of the SCTID to identify the most likely correct match. Semantic tags indicate where a concept fits into the medical hierarchy (i.e., disorder, finding, morphological abnormality, body structure, person, organism, or specimen). Since our primary goal is to identify diagnoses, we selected the SCTID in the following order: disorder, finding, morphological abnormality, and organism. This approach is corroborated by the observation that for all situations in which the matching of a case’s correct diagnosis returned only one SCTID (which happened in 95.6% of cases), these SCTIDs where overwhelmingly disorders (96.8%), followed by findings (2.0%), morphological abnormality (1.0%), and organism (0.2%).

We first applied this pipeline to the correct diagnoses of all the 1,572 cases. After normalizing the correct diagnoses, our approach could exactly match 1,333 (84.8%) correct diagnoses to an SCTID (SI Appendix, Fig. S2A). For these cases, we thus can be certain that we have identified the correct solution according to SNOMED CT. In the remainder, we focus on this set of 1,333 cases. The conservative approach of using perfect matching assures that we do not introduce errors when assigning a (correct) diagnosis to a SCTID. Next, we applied our approach to the diagnoses provided by the users who solved these 1,333 cases: 41,242 (out of 47,772; 86,9%; SI Appendix, Fig. S2B) could be exactly matched to a SCTID. The remaining 13.1% remained unidentified and were discarded from the analyses.

Having mapped the correct diagnoses and the users’ diagnoses into an integrated knowledge graph allows us to automatically aggregate users’ diagnoses and test how collective diagnoses compare to individual ones in terms of diagnostic accuracy. For each of the 1,333 cases, we implemented the following procedure. We considered groups of varying sizes (1 to 10). For each group size, we created all possible unique groups. For each of these groups, we determined the collective support for each of the unique SCTIDs provided by the group members using three aggregation rules (where $r$ is the rank of a diagnosis): $1/r$, $1/r^2$, and equal-weighting rule. The $1/r$ rule (Fig. 1C) weighs a diagnosis by the inverse of $r$: The first diagnosis provided by a user ($r=1$) receives a score of 1, the second one ($r=2$) a score of $1/2$, etc. The $1/r^2$ rule down-weighs diagnoses lower in the rank order more heavily (e.g., second-ranked diagnosis receives a score of $1/4$). The equal-weighting rule weighs all diagnoses equally (i.e., independent of order).

Within each group–and for each scoring rule—we summed the score for each of the unique SCTIDs provided by the group members and ranked the SCTIDs from the highest to lowest score. In case of tied scores, we ordered SCTIDs according to their semantic tags (in the same order as used in the concept mapping step, i.e., disorder, finding, morphological abnormality, and organism). If SCTIDs were still tied, we randomized the order within the respective tied SCTIDs. Finally, we determined whether the correct SCTID was present in the top 1, 2, or 3 diagnoses in the collective ranking.

Fig. 2 presents the results of this automated aggregation procedure, showing the average performance at each group size across all cases. Increasing the number of group members increases the diagnostic accuracy, that is, the likelihood that the correct diagnosis is present among the top 1, top 2, or top 3 of the collective ranking. For example, the likelihood that the correct diagnosis is present in the top 3 of the collective ranking increases from 46% for singletons to 76% for groups of 10 diagnosticians under the $1/r$ rule (Fig. 2, Left). The other two aggregation rules also lead to an increase in diagnostic accuracy with increasing group size (Fig. 2, Center and Right). There is, however, a difference in how much the diagnostic accuracy is increased. The equal-weighing rule (Fig. 2, Right) generally performed worse than both other rules, suggesting that the rank order of a diagnosis in a user’s diagnosis positively predicts diagnostic accuracy. SI Appendix, Fig. S3 shows that this indeed was the case: The diagnosis ranked first by a user was much more likely to be correct than lower-ranked diagnoses. These results indicate that it is important to give more weight to first-ranked diagnoses (as compared to equal weighing) but that the exact strength of this upweighing is less important. In the following, we will focus on the $1/r$ rule.

Next, we investigated whether improvements in diagnostic accuracy are robustly present across different medical specialties of cases (only considering medical specialties with more than 10 cases). Fig. 3 shows the diagnostic accuracy for different group sizes per medical specialty, showing that the improvement in diagnostic accuracy with group size is robustly found across medical specialties. SI Appendix, Fig. S4 shows that the same holds across different chief complaints of cases. Next, we compared different tenure levels. Of the 13,330 unique diagnoses, 3,054 were given by medical students (23%), 1,352 by interns (10%), 5,340 by residents (40%), 179 by fellows (1%), and 3,405 by attending physicians (26%). We compared the performance of small groups across the three most prevalent tenure levels. We considered only cases which were completed by at least three medical students, three residents, and three attending physicians ($n=62$ cases). Note that such a strict within-cases comparison is required as a between-cases comparison may be confounded with self-selection of users into cases. We used the same simulation procedure as described earlier. Fig. 4 shows that small groups outperformed single diagnosticians across all tenure levels. Attending physicians performed slightly better than medical students and residents when considering whether the correct diagnosis was ranked first, but not when considering the top 2 or top 3 diagnoses. SI Appendix, Fig. S5 shows the results when including all cases which were completed by at least two medical students, two residents, and two attending physicians (n = 450 cases) showing largely similar results. To summarize, combining the independent diagnoses of multiple diagnosticians robustly increases diagnostic accuracy across medical case specialties, cases’ chief complaints, and diagnosticians’ tenure level.

Besides linking the correct diagnoses and the users’ diagnoses to existing concepts in SNOMED CT for the automatic aggregation of identical concepts, our knowledge engineering approach also allows extracting and capitalizing on the interrelationships between concepts in the knowledge graph. SNOMED CT concepts are organized in a polyhierarchy, a graph structure whereby concepts (a.k.a., nodes) are connected to (one or multiple) supertype “parent” and/or subtype “child” concepts (Fig. 5A). Our knowledge graph incorporates these semantic relations, so that a diagnosis without an identical match to the correct diagnosis can be associated with a parent or child concept of the correct diagnosis. Such a diagnosis is, typically, more relevant than a diagnosis that is neither a parent or child because the former is closer to the correct solution—in terms of network path distance—and has a higher likelihood of implying similar (or even identical) treatment recommendations (35). Therefore, we next explored diagnostic accuracy when exploiting these interrelationships. Fig. 5B–D shows how diagnostic accuracy scales with group size when we consider a diagnosis correct when it is either a i) direct match with the correct SCTID (as reported above), ii) child concept of the correct SCTID, or iii) parent concept of the correct SCTID. This approach substantially boosts diagnostic accuracy across all group sizes, as compared to only considering identical matches. This indicates that users’ frequently reported parent and/or child concepts of the correct diagnoses. SI Appendix, Fig. S6 shows how often users reported parent and child concepts of the correct diagnosis, showing that both parent and child concepts appeared regularly in single users, and increasingly so in higher-ranked positions, explaining the increased performance when considering these as correct responses.

R.H.J.M.K., S.M.H., and V.T. designed research; R.H.J.M.K., A.G.N., and V.T. performed research; G.B. contributed new reagents/analytic tools; R.H.J.M.K., A.G.N., and A.R. analyzed data; and R.H.J.M.K., A.G.N., A.R., G.B., S.M.H., and V.T. wrote the paper.

G.B. reports having received personal fees from the Human Diagnosis Project outside the submitted work and is a spokesperson of the Human Diagnosis Project.