Skip to main content
SpringerLink
Log in
Book cover

From Research to Practice in the Design of Cooperative Systems: Results and Open Challenges pp 81–96Cite as

Harvesting Collective Agreement in Community Oriented Surveys: The Medical Case

  • Conference paper
  • First Online:

Abstract

The chapter discusses the role of simple and lightweight Web-based systems in promoting a different approach to the externalization of practice-related knowledge within communities of professionals. This approach exploits common online questionnaire systems to collect the preferences of large numbers of domain experts to interesting paradigmatic work cases and proposes a statistically sound evaluation of these responses to evaluate the agreement reached within the community. We tested this approach in a case study that involved a large international medical association, that we chose as an example of a large and highly distributed community of expert professionals; in this study we challenged more than 1,000 surgeons about some border-line clinical cases where tacit notions based on life-long practice and situated experiences coexist (and sometimes clash) with scientific evidences drawn from the specialistic literature. We make the point that a sound evaluation of the collective agreement is a necessary precondition to use such lean Web-based tools in bottom-up knowledge elicitation initiatives. To this aim, existing measures of agreement and survey-related heuristics can be exploited to get a more precise picture of the “opinion of the many” in collective settings like communities of practice.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.mmslists.com/.

  2. 2.

    http://www.nhs.net.

  3. 3.

    Obviously a doctor does not choose a treatment by chance; but it is the doctors that are (should be) recruited by chance.

  4. 4.

    Not to be confused with the Cohen Kappa, suitable for multi-case two-rater assessments.

  5. 5.

    In this latter case we compared the values of Chi with the Kendall’s coefficient of concordance, which is a normalized score between 0 and 1 as the above mentioned Alpha and Kappa.

  6. 6.

    These are, respectively, evidences of level II-3 and III or level C and D according to the evidence ranking developed by either the U.S. Preventive Services Task Force or the Oxford Centre for Evidence-based Medicine.

  7. 7.

    The logistic regression model we obtained is represented by the function y(x) = a/b + ce-ax with \(a \simeq 1.04,\,b \simeq 0.004,\,c \simeq 0.06\) and it is indicated with P(t) in Fig. 6.2.

  8. 8.

    This holds in the assumption that responses from the second turn are representative of the non-responses, and that people that can get convinced by a single reminder end up by exhibiting similar opinions to those that, conversely, are refractory to any reminder at all.

  9. 9.

    y′ = abce cx/(b + e cx)2.

References

  1. Aday, L.: Designing and Conducting Health Surveys. Jossey-Bass, San Francisco (1996)

    Google Scholar 

  2. Alstyne, M.V., Brynjolfsson, E.: Global village or cyber-balkans? Modeling and measuring the integration of electronic communities. Manag. Sci. 51(6), 851–868 (2005)

    Article  MATH  Google Scholar 

  3. Atkins, D., Eccles, M., et al.: Systems for grading the quality of evidence and the strength of recommendations: critical appraisal of existing approaches. BMC Health. Serv. Res. 4(1), 38 (2004)

    Article  Google Scholar 

  4. Barclay, S., Todda, C., Finlayb, I., Grande, G., Wyattc, P.: Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of gps. Fam. Pract. 19(1), 105–111 (2002)

    Article  Google Scholar 

  5. Braithwaite, D., Emery, J., de Lusignana, S., Sutton, S.: Using the internet to conduct surveys of health professionals: a valid alternative? Fam. Pract. 20(5), 545–551 (2003)

    Google Scholar 

  6. Burns, K., Duffett, M., Kho, M., Meade, M.: A guide for the design and conduct of self-administered surveys of clinicians. Can. Med. Assoc. J. 179(3), 245–252 (2008)

    Article  Google Scholar 

  7. Cabitza, F., Simone, C., Sarini, M.: Knowledge artifacts as bridges between theory and practice: the clinical pathway case. In: KMIA’08: Proceedings of the International Conference on Knowledge Management in Action. Held in Conjunction with the 20th IFIP World Computer Congress, 7 Jan 2008, Milan, Italy (2008)

    Google Scholar 

  8. Dillman, D.: Mail and Internet Surveys: The Tailored Design Method, 2nd ed. Wiley, New York (2000)

    Google Scholar 

  9. Dobrev, A., Haesner, M., Hüsing, T., Korte, W., Meyer, I.: Benchmarking ict use among general practitioners in Europe. Tech. rep., European Commission—Information Society and Media Directorate General (2008)

    Google Scholar 

  10. Feinstein, A.R., Cicchetti, D.V.: High agreement but low kappa: I. the problems of two paradoxes. J. Clin. Epidemiol. 43(6), 543–549 (1990)

    Article  Google Scholar 

  11. Ferlie, E., Wood, M., Fitzgerald, L.: Some limits to evidence-based medicine: a case study from elective orthopaedics. Qual. Health. Care. 8, 99–107 (1999)

    Article  Google Scholar 

  12. Gandhi, R., Tsvetkov, D., Davey, J., Mahomed, N.: Survival and clinical function of cemented and uncemented prostheses in total knee replacement. J. Bone. Joint. Surg. 91-B(7), 889–895 (2009)

    Article  Google Scholar 

  13. Grava-Gubins, I., Scott, S.: Effects of various methodologic strategies. Can. Fam. Physician. 54(10), 1424–1430 (2008)

    Google Scholar 

  14. Gwet, K.: Handbook of Inter-rater Reliability. STATAXIS Publishing Company, Gaithersburg (2001)

    Google Scholar 

  15. Hamilton, M.B.: Online survey response rates and times. background and guidance for industry. Technical report, Ipathia, Inc./SuperSurvey (2009)

    Google Scholar 

  16. Handoll H.H.G., Hanchard N.C.A., Goodchild L.M., Feary J.: Conservative management following closed reduction of traumatic anterior dislocation of the shoulder. Cochrane. Database. Syst. Rev. 25(1) (2006). (Art. No.: CD004962, doi:10.1002/14651858.CD004962.pub2)

    Google Scholar 

  17. Hayes, A.: Statistical Methods for Communication Science. Lawrence Erlbaum Associates, Inc., Mahwah (2005)

    Google Scholar 

  18. Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1(1), 77–89 (2007)

    Article  Google Scholar 

  19. Headrick, D.R.: When Information Came of Age: Technologies of Knowledge in the Age of Reason and Revolution, pp. 1700–1850. Oxford University Press, New York (2000)

    Google Scholar 

  20. Heide, L.: Punched-card Systems and the Early Information Explosion, pp. 1880–1945. Johns Hopkins University Press, Baltimore (2009)

    Google Scholar 

  21. Hiltz, S., Turoff, M.: Network Nation: Human Communication via Computer (Revised Edition). The MIT Press, Cambridge (1993)

    Google Scholar 

  22. Itoi, E., Hatakeyama, Y., et al.: Immobilization in external rotation after shoulder dislocation reduces the risk of recurrence. a randomized controlled trial. J. Bone. Joint. Surg. Am. 89(10), 2124–2131 (2007)

    Article  Google Scholar 

  23. Jacobs W, Clement D.J., Wymenga A.A.B.: Retention versus sacrifice of the posterior cruciate ligament in total knee replacement for treatment of osteoarthritis and rheumatoid arthritis. Cochrane. Database. Syst. Rev. 19(4) (2005). (Art. No.: CD004803, doi:10.1002/14651858.CD004803.pub2)

    Google Scholar 

  24. Jones, J., Hunter, D.: Consensus methods for medical and health services research. BMJ. 311, 376–380 (1995)

    Article  Google Scholar 

  25. Kane, B., Luz, S.: Achieving diagnosis by consensus. Comput. Support. Coop. Work 18(4), 357–392 (2009)

    Article  Google Scholar 

  26. Kapetanios, E.: Quo vadis computer science: from turing to personal computer, personal content and collective intelligence. Data. Knowl. Eng. 67(2), 286–292 (2008)

    Article  Google Scholar 

  27. Kellerman, S., Herold, J.: Physician response to surveys. A review of the literature. Am. J. Prev. Med. 20(1), 61–67 (2001)

    Article  Google Scholar 

  28. Kongsved, S., Basnov, M., Holm-Christensen, K., Hjollund, N.: Response rate and completeness of questionnaires: a randomized study of internet versus paper-and-pencil versions. J. Med. Internet. Res. 9(3), e25 (2007)

    Google Scholar 

  29. Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. Sage, Thousand Oaks (2004)

    Google Scholar 

  30. Landis, J., Koch, G.: The measurement of observer agreement for categorical data. Biometrics. 33, 159–174 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  31. Leece, P., Bhandari, M., Sprague, S., Swiontkowski, M.: Internet versus mailed questionnaires: a controlled comparison. J. Med. Internet. Res. 6(4), e39 (2004)

    Google Scholar 

  32. Lipton, R., Liberman, J., Cutrer, F., Goadsby, P.: Treatment preferences and the selection of acute migraine medications: results from a population-based survey. J. Headache. Pain. 5(2), 123–130 (2004)

    Article  Google Scholar 

  33. Marx, R.G., et al.: Beliefs and attitudes of members of the American academy of orthopaedic surgeons regarding the treatment of anterior cruciate ligament injury. Arthroscopy. 19(7), 762–770 (2003)

    Article  Google Scholar 

  34. McColl, E., Jacoby, A., Thomas, L., Soutter, J.: Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health. Technol. Assess. 5(31), 1–256 (2001)

    Google Scholar 

  35. Mosca, L., et al.: National study of physician awareness and adherence to cardiovascular disease prevention guidelines. Circulation. 111, 499–510 (2005)

    Article  Google Scholar 

  36. Parasuraman, A.: More on the prediction of mail survey response rates. J. Mark. Res. 19(2), 261–268 (1982)

    Article  Google Scholar 

  37. Porter, T.M.: Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton University Press, Princeton (1996)

    Google Scholar 

  38. Potter, W.J., Levine-Donnerstein, D.: Rethinking validity and reliability in content analysis. J. Appl. Commun. Res. 27, 258–284 (1999)

    Article  Google Scholar 

  39. Randelli, P., Cabitza, F., Arrigoni, P., Cabitza, P., Ragone, V.: Current practice in shoulder pathology: results of a web-based survey among a community of 1,084 orthopedic surgeons. Knee. Surg. Sports. Traumatol. Arthrosc. 20(5), 803–815 (2012). (doi:10.1007/s00167-011-1673-z)

    Google Scholar 

  40. Randolph, J.J.: Free-marginal multirater kappa: An alternative to fleiss’ fixed-marginal multirater kappa. In: Proceedings of the Joensuu University Learning and Instruction Symposium 2005, Joensuu, Finland, 14–15 Oct (2005)

    Google Scholar 

  41. Scheibel, M., Kuke, A., Nikulka, C., et al.: How long should acute anterior dislocations of the shoulder be immobilized in external rotation? Am. J. Sports. Med. 37(7), 1309–1316 (2009)

    Article  Google Scholar 

  42. Seguin, R., Godwin, M., MacDonald, S., McCall, M.: E-mail or snail mail? Randomized controlled trial on which works better for surveys. Can. Fam. Physician. 50(5), 414–419 (2004)

    Google Scholar 

  43. Sobal, J., Ferentz, K.: Assessing sample representativeness in surveys of physicians. Eval. Health. Prof. 13, 367–372 (1990)

    Article  Google Scholar 

  44. Thorpe, C., Ryan, B., Burt, A., Stewart, M., Brown, J.: How to obtain excellent response rates when surveying physicians. Fam. Pract. 26(1), 65–68 (2009)

    Article  Google Scholar 

  45. Van De Belt, T.H., Engelen, L.J., Berben, S.A., Schoonhoven, L.: Definition of health 2.0 and medicine 2.0: a systematic review. J. Med. Internet. Res. 12(2), e18 (2010)

    Google Scholar 

  46. VanGeest, J., Johnson, T., Welch, V.: Methodologies for improving response rates in surveys of physicians: a systematic review. Eval. Health. Prof. 30(4), 303–321 (2007)

    Article  Google Scholar 

  47. Webster, B.S., Courtney, T.K., et al.: Physicians’ initial management of acute low back pain versus evidence-based guidelines. Influence of sciatica. J. Gen. Intern. Med. 20, 1132–1135 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Università degli Studi di Milano-Bicocca, Viale Sarca 336, 20126, Milano, Italy

    Federico Cabitza

Authors
  1. Federico Cabitza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Federico Cabitza .

Editor information

Editors and Affiliations

  1. LIG, University of Grenoble, rue de la Chimie 110, St Martin d'Herès, 38400, France

    Julie Dugdale

  2. G-SCOP, University of Grenoble, rue de la Physique 230, Grenoble, 38041, France

    Cédric Masclet

  3. Xerox Research Group Europe, chemin de Maupertuis 6, Meylan, 38240, France

    Maria Antonietta Grasso

  4. G-SCOP, University of Grenoble, ave Felix Viallet 46, Grenoble, 38031, France

    Jean-François Boujut

  5. Institute of Political Study, rue de Saporta 25, Aix-en-Provence, 13625, Bouches-du-Rhône, France

    Parina Hassanaly

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag London

About this paper

Cite this paper

Cabitza, F. (2012). Harvesting Collective Agreement in Community Oriented Surveys: The Medical Case. In: Dugdale, J., Masclet, C., Grasso, M., Boujut, JF., Hassanaly, P. (eds) From Research to Practice in the Design of Cooperative Systems: Results and Open Challenges. Springer, London. https://doi.org/10.1007/978-1-4471-4093-1_6

Download citation

  • DOI: https://doi.org/10.1007/978-1-4471-4093-1_6

  • Published:

  • Publisher Name: Springer, London

  • Print ISBN: 978-1-4471-4092-4

  • Online ISBN: 978-1-4471-4093-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation