Introduction

Bordons et al. (2004) demonstrated the usefulness of bibliometric analyses to detect trends in the research of a therapeutic drug including the evolution of drug studies over time. In the assessment of success of new analgesics over the past 50 years (Kissin 2010) we observed a difference in the publication response to a new drug between biomedical journals in general and the top journals of the same category: the number of published articles on a drug increased (or declined) more rapidly in the top journals. The assessment of a new drug is a very slow process. The approval of a regulatory agency (Food and Drug Administration) is only the first step. The true drug assessment usually comes with the confirmation of their effectiveness by meta-analyses (the process of combining results of many clinical studies to draw conclusions about therapeutic value of a drug). The correct conclusion on a drug’s value reached via this process can take 10–20 years or even longer (Kissin 2010). We suggest that the relatively rapid publication response to a new drug in the top biomedical journals measured by a bibliometric index could have an important predictive value. Here we present a publication indicator based on the initial predominancy of publications in top biomedical journals, the Top Journals Selectivity Index. This indicator represents the ratio between the number of all types of articles in the top 20 biomedical journals and the number of articles in all (>5,000) journals covered by Medline, both over 5 years after a drug’s introduction. We assess this index in relation to its predictive value in the field of new analgesics. With this aim the correlation of this index with the drug’s success score based on the novelty of its molecular target, its analgesic efficacy, and the degree of response by the pharmaceutical market was measured.

Methods

The information on new analgesics was based on data obtained from the Food and Drug Administration (FDA) web site for approved drug products (http://www.fda.gov/Drugs). The selection of analgesics was limited to those approved by FDA during the period 1985–2009. They include New Molecular Entities (NME) developed as analgesics and also drugs developed for non-pain indications with subsequent FDA approval for the treatment of pain as an additional indication. The National Library of Medicine’s PubMed Web site (www.ncbi.nlm.nih.gov/PubMed) was used to count the number of articles in English published during certain time intervals. Specific keywords were selected according to the name of an analgesic in addition to using PubMed Medical Subject Heading (MeSH) terms. Terms added to the name of a drug were “pain” OR “headache disorder” [MeSH], OR “migraine.” Boolean operations were used in which variables were the selected keywords, years of publications, and type of publications.

The Top Journals Selectivity Index (TJSI), the ratio between the number of all types of articles in the top 20 biomedical journals and the number of articles in all (>5,000) biomedical journals covered by Medline, both over the 5 years since a drug’s introduction (referred to hereafter as “Top 20 Journals” and “All Journals,” respectively). The selection of the Top 20 journals was based on two factors—the rank of a journal sorted by the Impact Factor and by the specialty area related to the treatment of pain. The rank of journals sorted by the Impact Factor was determined on the basis of data presented by Journal Citation Reports for year 2008 (http://science.thomsonreuters.com). The areas related to the treatment of pain were: medicine in general and two specialty areas with specific interest in analgesic drugs—neurology and pharmacology. In each of these three areas only journals with highest rank (according to Impact Factor) were selected. Ten journals were selected for medicine in general category, six for neurology, and four for pharmacology. The selection was made by the author. As a result the following journals were included: N Engl J Med, JAMA, Nature, Nat Rev Drug Discov, Lancet, Science, Nat Med, Annu Rev Neurosci, Nat Rev Neurosci, Pharmacol Rev, Ann Intern Med, J Clin Invest, Nat Neurosci, Brit Med J, Trends Neurosci, Ann Neurol, Brain, Proc Natl Acad Sci USA, Trends Pharmacol Sci, J Pharmacol Exp Ther. The ratio of the number of all types of articles in the Top 20 Journals to the number of articles in All Journals covered by Medline (during the 5 years since the beginning of clinical publications on a drug) was expressed using two methods: as a percentage, and as a multiple of a projected ratio, obtained by division of the number of articles in the Top 20 Journals by the number of articles in All Journals for the general term “drug” and additional terms (“pain” OR “headache disorder” OR “migraine”) limited to the same year range as for the analgesic in question.

The Top Journal Selectivity Index was compared with two other indices of publication output. One was the number of all types of articles presented in Medline (PubMed web site), the other included only articles covering randomized controlled trials (RCT). The relationships of each of the three publication output indices with the measure of success (success score) in the development of new analgesics were determined.

The success in the development of analgesic drugs was measured by a scoring system based on the following criteria: novelty of the molecular target, the drug’s analgesic efficacy, and the degree of response to a new drug by pharmaceutical market. Novelty was assessed according to the level of advancement in molecular targeting, graded on the following scale: (1) An incremental improvement of an existing drug without the change of molecular target, 0 points; (2) A novel modification of the existing molecular target, 1 point; or (3) A completely novel molecular target, 2 points (Kissin 2010; Hill 2006).

Analgesic efficacy was assessed using the magnitude of pain relief as the main component, with strength of the conclusion on pain relief (ultimate proof of effectiveness) and universality of the analgesic effect (acute and chronic pain) as two additional components. The magnitude of effect was graded from 0 to 3, using a scale frequently employed to estimate analgesic effectiveness: if studies on a drug used as an index the number of patients who have achieved at least 30% pain relief—1 point; at least 50% pain relief—2 points; and pain-free response—3 points.

The ultimate proof of effectiveness was graded on the basis of the following three key elements: quality, quantity, and consistency. Quality is the extent to which the identified studies minimize the opportunity for bias (synonymous with validity). Quantity is the number of studies and subjects included in those studies. Consistency is the extent to which findings between different studies are similar (Ebell et al. 2004). Two levels of conclusion were used: inconsistent or limited-quality evidence—no points; consistent and good-quality evidence—1 point. The magnitude of pain relief and strength of the conclusion were assessed on the basis of published meta-analyses. Two positive meta-analyses were required to make the determination regarding the magnitude of effect and the strength of the conclusion. The assessment of universality of the analgesic effect was based on the capacity of a drug to provide clinically important effect in acute and/or chronic pain. One point was added only if a drug was effective in both acute and chronic pain.

The response of the pharmaceutical market was assessed by the introduction of similar drugs acting on the same molecular target as the original drug, a so-called drug “following”. If only one NME similar to the original drug was marketed the response was graded as 1 point, if more than 1–2 points. The analgesics evaluation was performed by the author who is a physician (pain management) and pharmacologist.

The closeness of the relationship between the analgesics success score and the indices of publication output was determined with the use of the Pearson correlation coefficient. The relationships were declared statistically significant if p was < 0.05.

Results

Fourteen drugs were selected for analysis. Four of them (gabapentin, duloxetine, topiramate, and valproate) were developed for indications other than pain, but were subsequently approved by the FDA for the treatment of pain. The publication output related to the assessed analgesics is presented in Table 1. It is limited to a 5-year period after publication of the first clinical trial (with the time interval for each of the analgesics indicated in the first column). This table has data on two indices—all types of articles and RCT articles. The third index, the Top Journals Selectivity Index, is presented in Table 2. This table contains 10 drugs because only analgesics with at least one article in the Top 20 Journals category and more than 10 articles in the All Journals category were selected. The Top Journal Selectivity Index (presented as an actual ratio and in the form of a multiple of the projected ratio) is in the Table. Sumatriptan has the highest score, 5.14. The score for zolmitriptan was 1.09, and five other second-generation triptans (not in the table) had scores between 0.82 and 0.89. Details of the sumatriptan publication output for the four 5-year periods are presented in Fig. 1. It presents the Top Journal Selectivity Index as a simple ratio of the number of articles in the Top 20 Journals to the number of articles in All Journals. This ratio was 21.1 during the 1990–94 period (it was 5.14 when presented as a multiple of the projected ratio in Table 2).

Table 1 Number of articles during 5-year period after publication of first clinical trial
Full size table
Table 2 Top journals selectivity index
Full size table
Fig. 1
figure 1

Sumatriptan-related publication output over four 5-year periods. a Number of all types of articles during each of the four periods in the top 20 biomedical journals. b Number of articles in all journals (>5,000) covered by Medline. c Ratio (%) of the number of articles in top 20 journals to the number of articles in all journals

Full size image

To test the indices of publications output as possible bibliometric predictors of drug success the scoring system described above was used and the results of its implementation is presented in Table 3. Only one of analgesics presented in the table, ziconotide, has a mechanism associated with a novel molecular target: the N-type voltage-sensitive calcium channel. Two other analgesics, sumatriptan and celecoxib, were developed as drugs with a novel modification of molecular targets arising from existing analgesics. Sumatriptan mimics the desirable effects of serotonin by acting on the 5-hydroxytryptamine 1B/1D receptor. Celecoxib selectively inhibits the cyclooxygenase (COX)-2 isoform of COX. As a result, only three drugs received points for the novelty of molecular target: ziconotide—2 points; sumatriptan and celecoxib—1 point each. For the category of magnitude of pain relief, remifentanil, sumatriptan, and zolmitriptan received 3 points each. All three analgesics were investigated in many studies with pain-free response as an index. Two drugs, pentosan and topical lidocaine, received 1 point each because they provided pain relief that was measured only with the use of response level close to 30%. With all other analgesics the common level of response was at least 50% pain relief, and each received 2 points for the magnitude of effect.

Table 3 Assessment of success in development of new analgesics
Full size table

The response of the pharmaceutical market by introduction of an NME similar to the original drug and acting on the same molecular target was graded with 2 points for sumatriptan (6 additional triptans) and one point for gabapentin (due to marketing of the similarly acting pregabalin). Celecoxib was not graded in this category because other selective COX-2 inhibitors (rofecoxib and valdecoxib) were withdrawn from the market. The total score characterizing the success in the development of a new analgesic is presented in the last column of Table 3. Sumatriptan received the highest score (7) and pentosan and topical lidocaine the lowest (1).

Figure 2 reflects the results of testing the relationship between the measure of success in development of new analgesics (success score) and the three indices of publication output: number of all types of articles presented in Medline, number of articles on RCT, and Top 20 Journals Selectivity Index. The figure indicates that although there was some tendency for positive relationships between the “all articles” (or “RCT articles”) index and success score, these indices did not reach statistical significance (Fig. 2b, c). At the same time, the strength of relationship between the Top Journals Selectivity Index and success score was quite impressive: r = 0.876, p < 0.001 (Fig. 2a). Even if “an outlier,” sumatriptan is not taken into account, the correlation coefficient continues to be high, 0.694 (p < 0.05).

Fig. 2
figure 2

The relationship between success score in the development of new analgesics and three indices of drug publication output. Along vertical axis, a Top Journals Selectivity Index (TJSI): the ratio between the number of all types of articles in the top 20 biomedical journals and the number of articles in all journals (>5,000) covered by Medline during 5-year period after the drug’s introduction (presented as a multiple of a projected ratio). b Number of articles on randomized controlled trials (during the 5-year period after the drug’s introduction) in all journals (>5,000) covered by Medline. c Number of articles on all types of publications (during 5-year period after drug introduction) in all journals (>5,000) covered by Medline. Along the horizontal axis, success score in the development of new analgesics data from Table 3. To topiramate, D duloxetine, P pregabalin, Zo zolmitriptan, Tr tramadol, R remifentanil, Zi ziconotide, C celecoxib, G gabapentin, S sumatriptan. The correlation coefficient (r) for Top Journal Selectivity Index and success score is 0.876 (p < 0.001). If “an outlier”, sumatriptan is not taken into account, it is 0.694 (p < 0.05)

Full size image

Discussion

This study assessed the development of analgesics introduced over the past 25 years. Fourteen were selected for quantitative assessment (Table 1). Two triptans are among them, one of which was sumatriptan. Only one second-generation triptan was included because they are generally quite similar. Zolmitriptan was chosen as their representative since publication output generated by it was somewhat greater than that by the other five triptans.

As indicated in Table 1, the publication impact of analgesics is quite variable. During the first 5-year period after publication of the first clinical trial, the number of articles (in >5,000 journals) for some of the drugs was very low. For example, with pentosan and topical lidocaine, the “all articles” category was less than 10 and the RCT category was 1 with pentosan and 3 with topical lidocaine. The numbers characterizing publications in the top 20 journals were of course much lower than in all journals (>5,000) covered by Medline. When the Top Journals Selectivity Index was calculated (Table 2), 4 drugs were excluded, either because they did not have any articles in the Top 20 Journals category or had fewer than 10 articles in the All Journals category.

Three diverse criteria were used for a drug’s quantitative assessment: novelty of molecular target, analgesic efficacy, and response by the pharmaceutical market. The contributions of these three criteria to the total assessment score were weighted differently: much more weight for analgesic efficacy (up to the maximum of 5 points for all 3 of its efficacy components taken together), and less for the two other criteria (up to the maximum of 2 points each). As a result, analgesic efficacy played a decisive role in the drug assessment. The strength of conclusion (ultimate proof of effectiveness) was based on the data presented in the meta-analyses. When the strength of conclusion was assessed as 0 (Table 3), it indicated that the level of conclusion was not of the highest degree. If the strength of conclusion received one point (then added to the points on the magnitude of effect) it meant that described effect was based on consistent results and good-quality evidence. The references for meta-analyses underlying this conclusion are placed in the same column as the strength of conclusion (in brackets). Only one analgesic with the strength of conclusion assessed as one point, remifentanil, did not have a meta-analysis. The use of remifentanil only in acute pain, its intravenous administration, and rapid onset and short duration of action are factors that have led to convincing results (Yarmush et al. 1997), and the conclusion was reached despite the absence of a meta-analysis.

The scores characterizing success in the development of a new analgesic (Table 3) ranged from one (pentosan and topical lidocaine) to seven (sumatriptan). The lowest score was mostly determined by the small degree of pain relief. The proportion of subjects with a specific percentage reduction in pain intensity is now commonly used to evaluate treatment efficacy. When a cutoff of 50% to dichotomized pain intensity is too high to measure the pain relief with an analgesic, a 25 or 30% cutoff is used. In several studies the minimal change in pain intensity noticeable to patients was determined to be 20% (Felson et al. 1998; Cepeda et al. 2003). When the magnitude of pain relief for an analgesic in Table 1 was graded as one point, it represents approximately 30%. Although low, this level of pain relief might be clinically important. The effect of topical lidocaine could be used as an example. The drug relieved pain intensity in postherpetic neuralgia only by 20–30% (Rowbotham et al. 1996) At the same time, a consensus meeting on pharmacological management of neuropathic pain recommended topical lidocaine as a first-line treatment for neuropathic pain (Dworkin et al. 2007). The comparisons of analgesic efficacy of various drugs in meta-analyses are usually indirect, using placebo as a common comparator (Stafford et al. 2009). As a result such comparisons are not reliable, only direct head-to-head comparisons between drugs can provide dependable information on comparative analgesic efficacy (Chou et al. 2009; Quilici et al. 2009). The absence of direct head-to-head comparisons makes it impossible to introduce into the analgesic assessment a category representing the advantage of one drug over another. Such comparison would not reach the level of consistent, good-quality evidence (Ebell et al. 2004).

The scoring system used in this study for the analgesics assessment is arbitrary and produces only categorical variables, which is an important limitation. The other limitation of this study is that due to the limited number of observations the sensitivity of different indicators of efficacy used for drug’s quantitative assessment was not tested. The possible alternative to our approach is either to canvas the subjective views of a large sample of relevant clinicians or to determine the sales figures for the assessed analgesics. However, these alternatives have their own problems. For example, sales figures depend not only on the number of pills sold and the size of drug doses, but also on the drug pricing influenced by the presence of generic drugs. Uncertainty in the interplay of these factors introduces important limitations for the reliable assessment of sales figures. As far as “the response of the pharmaceutical market” is concerned, it involves the interaction of sales and profits that makes the assessment even more difficult. The competitive response of pharmaceutical companies introducing “me too drugs” reflects companies’ assessment of a drug’s market potential. The scoring based on companies’ decisions in this regard is lacking in reliability, but has the advantage of diversity in the assessment approaches.

The assessment of relationships between the drug success scores and publication output in biomedical journals confirmed that the Top Journals Selectivity Index could be one of the indicators of drug success. The relationship between that index and the analgesics success score was positive and reliable (r = 0.876, p < 0.001). It was based on the analysis of relationships between success scores and TJSI of 10 drugs. Four drugs (ketorolac, pentosan, topical lidocaine, and valproac) were excluded from this analysis because they had no articles about them in the Top 20 Journals. If a score of zero (for TJSI) is given to these four drugs and the relationship between the index and the analgesic success score is based on 14 drugs, the relationship is not significantly different from that based on 10 drugs: r = 0.815, p < 0.001. TJSI is the ratio of the number of articles in the Top 20 Journals to the number of articles in ALL Journals covered by Medline (>5,000). If the number of articles in the Top 20 Journals is 0, the TJSI will be also 0 regardless of the number of articles in All Journals. For example, ketorolac and pentosan both have no publications in the Top 20 Journals; however, the number of articles in All Journals with these two drugs is quite different: 41 for ketorolac and only 7 for pentosan. When TJSI for both of them is calculated it has the same value—0. In this case, the measure is mostly a reflection of the lack of data for proper TJSI calculation and does not actually characterize the index.

Koenig (1982), who dealt with the relationship between the success rate of drugs of various pharmaceutical companies and different factors of research performance, defined a Composite Drug Output Index which served as a good predictor for past (or) future success. His finding of the high correlation between number of all published articles and expert judgment of the quality of pharmaceutical research is of particular interest. His data also indicated that the number of highly cited clinical articles correlated with drug research success (assessed by expert judgment) even better than the number of all clinical articles. Because the top biomedical journals selected for our index calculations have a much higher citation rate than the rest of the journals, the above mentioned better correlation rate for highly cited clinical articles compared to all clinical articles could serve as an indication going in the same direction as our conclusion.

The Top Journal Selectivity Index reflects the publication pattern during the first 5 years after a drug introduction. At the same time, the analgesic success score reflects a period much longer than the initial 5 years. It is based on meta-analyses of the results of multiple randomized controlled trials. The correct conclusion on a drug’s analgesic value reached via this process could take 10–20 years or even longer (Kissin 2010). The difference in the time-course of drug assessment based on the Top Journal Selectivity Index and the usual process of drug assessment associated with meta-analysis suggests that the index could have a predictive value. The predictive value of this index can probably be explained by the high quality of experts involved in the assessment of manuscripts evaluating new drugs in the top specialty journals. It remains to be seen that the predictive value of this indicator is also good for new drug development in the fields other than analgesics. In conclusion, a bibliometric indicator based on initial predominancy of publications in top biomedical journals, the Top Journals Selectivity Index, has a promise for the prediction of success in the development of new analgesics.