Category Archives: Critical Appraisal

Summary of PHTwitJC #2: Prostate-Specific Antigen Concerntration at age 60

Participants and followers of #PHTwitJC voted on their preferred topic to discuss and critique, from a selection of case-control papers put forward.  It was intended for this paper to be discussed on 28th August, however due to low participation it was re-run.

On Sunday September 11th #PHTwitJC discussed a paper by Vickers, A et al ((2010) Prostate specific antigen concentration at age 60 and death or metastasis from prostate cancer: case-control study).  The transcript of discussions can be accessed here.

Q1 – This paper uses a case-control study design – was this appropriate for this topic?

Participants agreed that a case-control design was appropraite as the outcome was suitably rare (death from prostate cancer).  Cases and controls were identified from a previous cohort study (otherwise known as a nested case-control study design); it was suggested that using a clearly defined closed cohort strengthened the design.

Q2 – Did recruitment for cases and controls affect selection bias?

It was unclear from the paper how participants were recruited in the original cohort study, therefore it is difficult to ascertain the probability of selection bias.  It was questioned whether a bias was created through the different inclusion criteria for cases and controls (i.e. cases were dead, but controls were all alive). This is likely to cause a sample selection bias within the control group by discounting those who were dead.  Participants discussed what the implications of this bias would be upon the effect size.

Q3 – What confounding factors could be present?  Have these been accounted for?

A number of potential confounding factors (such as socio-economic status (SES), ethnicity, family history and other health conditions) were not accounted for in this paper.

Q4 – Study concluded that “a single measure of PSA at age 60 is associated with a man’s lifetime risk of death from prostate cancer” – are these results valid?

Participants questioned whether the results of this study were clinically useful, for example

“Stats hold up but can you predict when someone needs treating ie at 60 or 80? Would treat at 60 prolong life?” (@TwitttwooSue)

Participants did not feel confident that a single measure can infer lifetime risk of death from prostate cancer, especially as the study does not consider other causes of death (sample selection bias).  It was noted that the authors used different thresholds for different parts of the analysis, which can be considered problematic.

Q5 – What are the PH implications of this study? How might these influence future practice/policy?

It was noted that other RCT evidence have demonstrated that prostate-specitic antigen can identify cancers, but it has not been shown to prolong life (all-casue mortality).  Although the authors stated that the findings should inform screening programmes, it was noted that this was not discussed in the paper.

A query was raised questioning why investment in research for evidence to support prostate cancer screening continues, despite evidence to date suggesting that prostate cancer is not suitable to a screening programme.  @carotomes wondered whether there was political weight attached to the research due to the lack of male screening programmes, however @TwittwooSue pointed out that the male-only screening programme for Abdominal Aortic Aneurysm (AAA) screening is due to be launched soon.

Overall it was felt that this paper provided insufficient evidence to support the use of PSA as an effective screening tool for prostate cancer.  #PHTwitJC participants agreed that this paper shouldn’t influence change in public health policy or practice.


CA Jan 2003 Past Paper: Antidepressant drugs and generic counselling

In this week’s study session we (5 registrars in the region who are preparing for the January exams) we discussed our critical appraisals of the following paper: Chilvers et al (2001) Antidepressent drugs and generic counselling for treatment of major depression in primary care: randomised trial with patient preference arms, BMJ

Q1 required candidates to write a structured abstract, which is pretty difficult to do after reading the whole paper, including the abstract… I shall attempt this next week!

Q2: Critically appraise the paper, paying particular attention to: the methods (including statistics), choice of outcomes, results, and conclusions drawn


  • Although GP practices were randomised, only patients who self-selected to join the randomisation group were randomised.  These patients tended to be less severely depressed and their characteristics may have differed from those refusing randomisation (such as levels of motivation, previous experience…etc)
  • GPs recruited participants which may have provided an element of recruitment bias.  Did GPs have preference for a particular treatment?  No details provided or discussion of ways to avoid recruitment bias.
  • Authors acknowledged recruitment to be difficult (and even adjusted their power calculation and study numbers as a result).  No baseline figures of number of patients approached/refused were provided… not assured this is a representative sample of the general population with depression.
  • Statistics: not all baseline characteristics were reported in table 1.  Chi-square and Fisher’s Exact test both appropriate for small sample size.


  • Paper title suggests an investigation of patients with diagnosed major depression, whereas throughout article reference is made to mild/moderate depression.  Furthermore depression is diagnosed by ‘research criteria’ for the study which limits it’s application to the real-world
  • Problematic use of the term “generic counselling” – what does that mean?  Full treatment detail is really important, especially if this paper hopes to inform commissioning of services.  Again – lack of information limits the real-world application
  • Also, although study details that GPs were provided with protocol for anti-depressents, no attempt to assess the efficacy of this, or monitor how GPs managed patients on anti-deps…etc GP consultations are in themselves interventions.  Many unknowns regarding treatment process that have important implications for considering the paper’s conclusions…
  • Paper assumed that if patient was well at 8 weeks and again at 12 months, they were in remission throughout that period.  Given the nature of depression, this is an unfounded assumption to make as most patients will experience a rise and fall of symptoms throughout their recovery.  Authors refers to MANY assumptions, which is suggestive that they did not have appropriately defined periods of follow up (nor, as mentioned above, did they assess frequency of GP consultations as part of treatment).


  • Reports no systematic different between proportion of patients asigned to different trial arms – however no test of hetergeneity for the baseline charactertics (age, sex, ethnicity)
  • Patients choosing counselling reported to do better compared to those randomised to counselling – but 95% confidence interval reported as 0.0 – 9.2 ….. this is fairly wide considering a mean difference of 4.6, plus a lower limit of 0.0… is this considered as ‘crossing zero’ and therefore increase the likelihood that this result could be due to change?


  • Reported conclusion suggested that “12 months after starting treatment, generic counselling is as effective as anti-depresants”… however I disagree as firstly) the term ‘generic’ counselling is meaningless, secondly) not clear what severity of depression was being treated through this trial, and thirdly) the authors have omited to specify which type of anti-depressants were used… the lack of specific details make it difficult to agree with this conclusion
  • Not confident that confounding factors were well accounted for e.g. counselling has a gender bias, self-selection vs. willing to be randomised may have an effect on time to remission due to personal motivation…
  • Article further concludes that both treatments (again, lacking specific details of what treatments were?) are effective (again, lacking specific details of treating what type of depression? and defined using criteria which is not applicable to general practices).

(NB: Ideally… I should be preparing my critical appraisal under timed conditions, and will do in future.)

Q3 A review group is being pressed to fund the expansion of counselling services.  What would be your response; what additional information is needed?

As a group, we struggled to identify a framework to structure the answer to this question… any ideas?

In order to review a funding application it would be important to have access to the following sources of information:

  • Defintion of the condition being considered (is the expansion specifically for patients with depression? Counselling can be used to address a number of problems)
  • Epidemiology of the condition (including local incidence and prevalence, so trends and local needs can be considered) and local projections
  • Literature review / NICE guidance to help inform how best to commission services for the condition and ensure the evidence base is known and applied
  • Full details of the current services available (including numbers attending services for this condition, waiting lists if available (as this would evidence unmet need))
  • Funding application ought to include cost-effectiveness analysis whereby a return on investment can be demonstrated, with clear timelines as to when that ROI would be expected

My response would be to firstly identify information needed (as above) and collate this into a report which has a particular focus on the evidence-base, local need + projections, and cost-effectiveness of counselling to address this condition.  It would be important to involve stakeholders in this process of collating information, but also to present the report findings to the group reviewing mental health services to ensure that any altered commissioning arrangements are based on evidence of need and consider the bigger public health picture

9/11 – Reflections from a Public Health Perspective

It’s September.  This is the month when the critical appraisal papers *generally* are chosen for the January Part A FPH Exam, *typically* from the BMJ… (so I’m making a mental-note to review papers which were published at the end of the month).  But this isn’t just any old September…

This September marks 10 years since the ‘9/11’ attacks.  On Tuesday September 11th 2001, four coordinated suicide attacks by al-Qaeda were carried out on the United States.  Reflections on this 10 year anniversary have started to creep onto all the news channels, and I assume these reflections will be sustained until the anniversary day (a week today). But how can a public health perspective contribute to such reflections…?

Handily, The Lancet have today published a series of articles which examine the health consequences of the 9/11 events.

The editorial “9/11 – Ten Years On” observed that the US governmental responce focused on defense, security and emergency preparedness.  Not only did this encourage fear and anxiety amongst the general population, it stigmatised many muslim communities and individuals.  Furthermore, health was pushed onto the backseat of political agendas, and the author suggests that:

“.. 9/11 was a huge opportunity cost for the health of the American people.”

As well as US domestic health effects, the 9/11 events have had international consequences.  One postitive outcome has been an increased post 9/11 commitment to global health encompassed by the US national security strategies.  The US recognised that investing in global health (amongst other aspects of development) had positive outcomes in terms of stability and security for their country. 

The 9/11 events, responses and consequences have provided a lot of reflections and learning for public health; from the micro (such as individual health protection, occupational health, suicide attacks), to the macro (health policies, emergency preparedness and reponse).

Articles of interest include:

“The events of 9/11 not only represent an example of a local act with global consequences, but also an instance where poverty and perceived injustice can contribute to catastrophic global instability and insecurity. It is now abundantly clear that human-made crises will, if not resolved decisively through politics and diplomacy, create the conditions for human-made disasters.”

Critical Appraisal of Prostate Screening #PHTwitJC Paper

In preparation for this weeks #PHTwitJC I thought it would be a good idea to 1) read the paper in it’s entirety, then 2) critically appraise the paper using the CASP framework for case-control designs.  My theory is the more familiar I am with the paper, the easier it will be to chair the discussions.

(Also – critical appraisal will be tested for within the Part A exam, plus it’s relevant to LO 2.14… so one piece of work can tick a few boxes.  And I’m hoping that practice will make it easier and quicker to do…).  Now for a prostate screening related cartoon:

Critical appraisal of Vickers et al (2010) Prostate specific antigen concentration at age 60 and death or metastasis from prostate cancer: case-control study, BMJ 2010;341:c4521

1 – Did the study address a clinically focused issue?

Yes.  Population = Swedish men (data taken at age 60 and followed-up outcomes at age 85).  Risk factor studied = PSA concerntration at age 60.  Study assessed association between PSA concerntration and outcome of death from prostate cancer, or prostate cancer metastatis.

2 – Did the authors use an appropraite method?

Prostate cancer in men is relatively high (estimate is that close to 40% of 70 year old men have detectable cancer in the prostate at autopsy), yet death from prostate cancer is relatively low.  Therefore the outcome of ‘death from prostate cancer’ is suitably rare for a case-control study design, as used in this study.  The key question of this paper was whether prostate cancer is likely to cause symptoms or shorten life, not whether a man has prostate cancer.

3 – Were the cases recruited in an acceptable way?

Cases were obtained from the ‘Malmo Preventive Project‘ Cohort (subjects were invited to participate in a broad health-screening programme, including a physical examination and a panel of laboratory tests – the objective of the project was to identify individuals with high-risk of cardiovascular disease and alcohol abuse).  As a large representative sample of the population, with health information and stored samples, on face value this appears to be a convenient and appropriate recruitment method.  The paper described the cohort as:

The cohort reported here includes 1167 men born in 1921, constituting 71% of the eligible population, who gave blood and completed medical history and lifestyle questionnaires in 1981-2

According to the MPP website link and numbers there was a large drop-out rate (as often the case in cohort studies); only 40% had completed the study in full.  I’m not sure what ‘eligible population‘ refers to – men born in Malmo in 1921?  Cohort population who had completed health data? Is ‘cohort reported here’ different from MPP?  Important to know precisely what population is being referred to.  Also – there is likely to be a healthy-participant selection bias as participants would have agreed to tests to be taken and to participate in the topic of the research (CVD and Alcohol).

Case selection: cases were defined precisely and verified by blinded/independent clinicians.  Number of cases selected was all meeting case definitions within cohort.  Case identification was reliant upon Cancer Registry records, which are reported to be very reliable in Sweden.  No power calculation was apparent.

4 – Were the controls recruited in an acceptable way?

We used a nested case-control design to analyse data from the cohort. Separate matches were conducted for each of the three study events: clinically diagnosed prostate cancer, prostate cancer metastasis, and death from prostate cancer.

Cases were matched 3:1 (which is excellent!) from the same cohort population (therefore the same healthy-participant selection bias applies).  For each case, controls were selected at random (precise randomisation method omited) from a group of men who were alive and event-free at the time of the event in the case.  They were matched on date of birth and date of base-line blood sample (with a 3 month window).

I guess three questions arise here. 1 – what impact would a healthy-participant selection bias have upon the study? Is this group different to the general population? This population is probably more likely to engage in any screening programmes, but do we know whether the incidence of life-threatening prostate cancer is comparable from this group to the general population?  2 – (and this may be a silly question, but) why did cases need to be alive at the time of the case-event? Does this not contradict the point of controls; for them to be exactly the same (date of birth, blood sample) apart from their outcomes (e.g. prostate cancer or metastatis is cases and diseases-free or death in controls).  What effect does exclusion of dead controls (at time of case-event) have upon study?  3 – (again, this may be a silly question, but) would it be realistic and/or useful at all to match cases and controls to PSA levels?

5 – Was the exposure accurately measures to minimise bias?

Case-control studies normally have a disadvantage of reliying upon historical records and memory/recall for exposure measurements.  However, this study design overcomes such problems as the exposure (PSA) was measured prior to the events.  Furthermore, the exposure samples were assayed in anticoagulated blood plasma (stored at −20°C) and re-tested and ‘determined as previously reported’.  Identical objective methods were used to assess PSA concerntration levels for both cases and controls.  It was not mentioned whether the re-test were done by someone blinded or independent to this investigation, but that would’ve increased the reliability of such concerntration measures.

The use of PSA as a valid marker for prostate cancer was discussed by the authors.

6 – What confounding factors have the authors accounted for? Were these accounted for in the design/analysis?

Main factors which concern me relate to the original cohort recruitment.  Although the authors of this study have reported the age and clinical exposure and outcomes of participants (both cases and controls), no consideration seems to have been given to their socio-economic status, ethnicity (men with african-heritage are more at risk), family history (risk doubles when prostate cancer present in immediate family), and other health conditions (such as men with diabetes mellitus appear to have a lower risk of developing prostate cancer).  (For further information on Epidemiology of Prostate Cancer go here)

None of these potential confounders were acknowledged or accounted for in this paper.

7 – What were the results of this study?

There were 43 cases of metastasis and 35 deaths from prostate cancer. Concentration of prostate specific antigen at age 60 was associated with prostate cancer metastasis (area under the curve 0.86,  95% confidence interval 0.79 to 0.92; P<0.001) and death from prostate cancer (0.90, 0.84 to 0.96; P<0.001)… Although only a minority of the men with concentrations in the top quarter (>2 ng/ml) develop fatal prostate cancer, 90% (78% to 100%) of deaths from prostate cancer occurred in these men. Conversely, men aged 60 with concentrations at the median or lower (≤1 ng/ml) were unlikely to have clinically relevant prostate cancer (0.5% risk of metastasis by age 85 and 0.2% risk of death from prostate cancer).

8 – How precise are the results?

Can’t fault reporting of ROC curves, tables, 95% confidence intervals with associated p-values.  I found the results quite straightforward and easy to understand.

*However*… one cannot rule out the effect of counfounders; the participant characteristics were not reported, acknowledged or discussed, nor adjusted for.

As well as considering the influence of participant characteristics within cohort, important to also consider charactertistics of non-responders/drop-outs… this is particularly important when drop-out is high (i.e. 60% in the MPP).  The author should not assume that all readers are familiar with the MPP study, and moreover they should report the charactertistics of their participants as this is highly relevant to the interpretation of their reported findings.

9 – Do you believe the results?

The results appear to be positive – it is easy to be swayed due to the positive correlation between PSA concerntration and prostate cancer outcome.  The author has reported a significant p-value, therefore the results are unlikely to be due to chance.  The effect of bias and confounding cannot be ruled out…

The study design and methods are well reported and objective, but the study has been let down in it’s reporting of participant (and non-participant) charactertistics.  Therefore even if the results were valid, one could not be sure whether these results are applicable to local and general populations.

10 – Can the results be applied to the local population?

Insufficient description of cohort population (aside from nationality and age) to make comparisons to local population.  Authors acknowledge further research on PSA and race would be useful, yet no description of the ethnic classifications is provided in this study.

11 – Do the results of this study fit with other available evidence?

Screening is associated with considerable overdiagnosis, and many men need to be screened to save one life.  PSA is a common screening tool for prostate cancer and this study seeks to stratify risk of adverse-event prostate cancer with PSA as meansured at age 60.