For Transplant Center Professionals

REVIEWING YOUR CENTER'S REPORTS

I understand SRTR releases the data on my center to review, prior to releasing the reports; where do I find that?

What is the process for the release of the program-specific reports (PSRs)? 

Where are the cohorts for the various tables in the PSR listed?

PROGRAM SPECIFIC REPORTS (PSR) METHODOLOGY- WAITLIST

Which patients are included when assessing a program’s waitlist mortality rate?

Are patients who are inactive for all or part of the 24-month period included in the calculation? 

What happens when a candidate’s condition deteriorates and he or she is removed from the waiting list? 

How long will a patient be followed in the Program Specific Reports?

How does SRTR count patient time?

What is the formula for patient years?

How long will a death on the waiting list continue to appear on a program’s report?

What is the formula for the waitlist transplant rate?

What is the formula for the waitlist mortality rate?

What is the formula for the waitlist expected transplant rate?

What is the formula for the expected waitlist mortality rate?

The waitlist models used to include one-year cohorts. It seems like it is two-year cohorts now?

Why aren’t P-Values included anymore in the model documentation?

Are candidates listed for multi-organ transplants included in the transplant rate and waitlist mortality calculations for each organ separately?

PROGRAM SPECIFIC REPORTS (PSR) METHODOLOGY- POSTTRANSPLANT

Where are the cohorts for the various tables in the PSR listed?

Some of our patients in the cohort aren’t showing a death or graft failure.

How does SRTR define a multi-organ transplant?

Are recipients of multi-organ transplants included in the calculations for posttransplant survival?

What is the historical rationale for excluding multi-organ transplants?

Why are heart-lung reports released, but not liver-kidney reports? Heart-lung transplants occur less frequently than liver-kidney. Are any changes planned?

How are retransplants handled when assessing outcomes in the program-specific reports?

Is a kidney transplant after a kidney-pancreas transplant considered a retransplant when assessing outcomes in the program-specific reports? 

If a candidate underwent transplant 10 years ago, are all subsequent transplants considered retransplants or only those that occur within a certain time period?

Why Does OPTN include Retransplant as a diagnosis on the TIEDI forms, but SRTR doesn’t consider it a true diagnosis in their risk adjustment models?

Are bilateral or en bloc transplants, counted as 1 or 2?

OFFER ACCEPTANCE TABLES

What constitutes an offer? Does an offer have to be a primary offer for a recipient at a transplant center, or does an offer count for those for patients who are further down the match run?

For the “hard-to-place” organs, does “over 50 offers” mean that there were more than 50 offers total, or that the organ was turned down over 50 times before it was accepted?

Is an "acceptance" only counted if an organ is transplanted?

When a patient is status I or has a MELD score of greater than 35, and a liver is accepted for transplant, the center will continue to receive offers for that recipient as the recipient is not removed from the list until after the transplant. These offers have to be declined. Are these also counted as declines?

Is organ offer acceptance data only for single organs?

Are Multi-organ transplant excluded from these numbers?

RISK ADJUSTMENT MODELS

What is risk adjustment?

I’m unable to see the risk adjustment models. The screen I see is blank (or grey).

Primary Diagnosis value is missing, although we entered “retransplant” on the TIEDI

You used to have a document the explained where the covariates come from on the TIEDI forms and I can’t find it now?

How is a “missing” value treated in the risk adjustments and why?

There are elements included in data sheets, but not in elements list in the risk adjustment models online?

Where are risk adjustment models from previous reporting periods?

Why do some adjustments seem counter-intuitive; previous malignancy, for example?

What is LS?

When are the risk-adjustment models rebuilt?

What is the difference between a “rebuild” and a “refit” of the risk adjustment models?

What is the difference between the Estimated and Expected probability?

Why isn’t the estimated probability shown the same percentage I get when I divide the observed failures by the transplants observed?


Reviewing Your Center's Reports

I understand SRTR releases the data on my center to review, prior to releasing the reports; where do I find that?

You will find your center’s Data Integrity Reports and Expected Survival Worksheets posted on the SRTR secure site designated for your center. If you do not have access, email srtr@srtr.org and they will guide you to your secure site administrator, whom will add you upon request.

What is the process for the release of the program-specific reports (PSRs) and the OPO-specific reports (OSRs)? 

PSRs and OSRs are released publicly in January and July of each year. The release cycle occurs in three primary phases:

  1. Data review period: SRTR uses data provided last day of September or March to create draft reports for programs to review during October 1-31 or April 1-30. Programs should look at their Data Integrity reports provided on the SRTR secure site to correct their OPTN data during this review cycle. (For information on how to get on to your program’s SRTR secure site, contact us at srtr@srtr.org.)
  2. Secure release: Following the data review period, SRTR uses OPTN data provided in November or May to create preview versions of the final Program Specific Reports that are then provided to programs in December or June. This is also when the Expected Survival Worksheets are provided on the SRTR secure site for centers to review their outcomes measures with the new risk adjustments applied. Programs also have the opportunity to provide comments to be appended to the public version of the report at that time.
  3. Public release: In January or July, programs have the opportunity to provide comments up to a month after the public release. SRTR posts any comments provided by the programs on our public website.

Where are the cohorts for the various Waitlist tables in the PSR listed?

We have them listed on the page of the SRTR public website here: PSR Reporting Timeline, and they are updated every time the new PSRs and OSRs are publicly released in January and July.

Program Specific Reports (PSR) Methodology- Waitlist

Which patients are included when assessing a program’s waitlist mortality rate?

When SRTR assesses a program’s waitlist mortality, we use a recent 24-month period (cohort). Any patient who was listed at the program before the close of the 24-month period and was known to be alive for at least 1 day of the 24-month period is considered in the period reported in the PSR. The waitlist mortality data that SRTR presents are meant to reflect mortality after listing, but before transplant. Therefore, SRTR continues to follow each patient, even those removed from the waiting list, with a few exceptions. The exceptions are: candidates who were removed from the list due to recovery (i.e., transplant is no longer needed because the patient’s condition improved), or who transferred to another transplant program. SRTR continues to follow recovered patients for a maximum of 60 days. Any death occurring during the 60 days would be attributed to the program. In summary, any patient who was listed by the transplant program before the close of the 24-month period and was alive and met the above inclusion criteria for at least 1 day during the 24-month period are included. No time after transplant is included in the calculation. The final calculation is the ratio of the total number of deaths observed divided by the total person-time in days observed in the 24-month reporting period.

Are patients who are inactive for all or part of the 24-month period included in the mortality calculation? 

Inactive time on the waiting list is treated no differently from active time. Deaths that occur during periods when the patient is active or inactive on the list are counted in that period.

Are patients who are inactive for all or part of the 24-month period included in the mortality calculation? 


Inactive time on the waiting list is treated no differently from active time. Deaths that occur during periods when the patient is active or inactive on the list are counted in that period.

What happens when a candidate’s condition deteriorates and he or she is removed from the waiting list? 

The waitlist mortality metrics that SRTR produces include deaths that may have occurred after removal from the waiting list (with the exceptions as noted above). Candidates who are removed from the list due to improved condition are followed for a maximum of 60 additional days. If a death occurs during that time, the death would be counted. If a candidate is removed from the waiting list for other reasons (other than undergoing transplant), time on the waiting list continues to accrue as if he or she remained on the list, and the candidate’s death would be counted in the calculation.

How long will a patient be followed in the Program Specific Reports?

As long as a patient is alive on the waitlist during a reported period, they will contribute to the reported statistics. First, they continue to be followed and accrue person time until transplant. They continue to be followed even if they were removed from the list due to recovery. SRTR continues to follow these patients for a maximum of 60 days post removal, unless death occurs in that 60 day window, and then the patient is followed only until death. The patient can be followed up-to the end of a reporting period, if the patient has been removed for reasons other than transplant, recovery or transfer. Finally, if none of the above occurred, the patient can be followed until death.

How does SRTR count patient time?

Since SRTR is reporting on one observation period equal to a two-year time-frame, we consider all patients on the waitlist at the beginning of an observation period and who were added any time after the beginning of the period. Patient time begins to accrue for a patient from the beginning of the period, if that patient was already on the list at the beginning, or it begins to accrue from the day the patient was added to the list. Patient years for a patient included in that reporting period stop at the end of the period, or if the following occurs: transplant, death or 60th day post-removal; however, If the patient has been removed for reasons other than transplant, recovery or transfer, that patient contributes to patient time up-to the end of that period. (See diagram for more detail.)

What is the formula for patient years?

Since candidates may be observed for all or for only part of a full year, person-years are reported. Person-years are calculated as the number of days the candidate was observed and converted to fractional years for each candidate. (The calculation includes all patients already on the waitlist at the start of the period, plus all patients added to the waitlist within that 1-year timeframe.)

So for SRTR calculations, let’s say there are two patients at your center. For this example we'll count days, and we only look at a 1 year timeframe.

Person No.         Total follow-up (in days)                 1 year

1                              182                                                  182/365           = .499

2                              300                                                  300/365           = .822

                                TOTAL                                                                        1.321

The transplant or mortality rate is calculated by dividing the number of events (waiting list deaths or transplants) by the person years. If there was one event, then  1 / 1.321= 0.757 events per person year.  Because events per 100 person years is reported, 0.757 is multiplied by 100 to get 75.7 events per 100 person years.

Events per person year are reported rather than events per candidate, because candidates are not all on the list for equal amounts of time.  Events per person year are reported rather than events per year, because programs have different numbers of candidates on their lists.  Using person years allows the metric to adjust for differences in the number of candidates and differences in the amount of waiting time for each candidate.

How long will a death on the waiting list continue to appear on a program’s report?

We report on the waitlist activity for candidates in a given cohort (two-year time-frame). Any events affecting the candidates in that cohort will be reported. The event (in this case a death), will be reported on in successive reporting periods until the candidate’s listing date falls out of the cohort. (See diagram)

What is the formula for the waitlist transplant rate?

The transplant rate is calculated by dividing the -number of transplants by the person years.

What is the formula for the waitlist mortality rate?

The waiting list mortality rate is calculated by dividing the number of deaths by the person years.

What is the formula for the waitlist expected transplant rate?

The SRTR uses models to predict the expected number of transplants for each candidate.  The expected transplant rate is the number of expected transplants divided by the person years.

What is the formula for the expected waitlist mortality rate?

The SRTR uses models to predict the expected number of deaths for each candidate.  The expected waiting list mortality rate is the number of expected deaths divided by the person years.

The waitlist models used to include one-year cohorts. It seems like it is two-year cohorts now?

Yes, SRTR rebuilt the waitlist models to be more predictive. The Fall 2017 PSR cycle was the first to produce results based on the new two-year cohort, for kidney, liver, heart and lung. Two-year cohorts allow us to compile more data to develop the more comprehensive and predictive risk adjustments. Intestine and pancreas models will be updated with the two year cohorts in the near future.

Why aren’t P-Values included anymore in the model documentation?

The goal of the models used in the PSRs is to produce the most predictive models, not make scientific inference about what factors are clinically important.  P-values are used to make inferential conclusions, so they are not relevant to the PSR models.   SRTR is in the process of shifting to a model development framework that emphasizes how well the models predict, rather than whether the predictors are significant.  The new modeling framework uses a penalized regression, and the penalty value is chosen to maximize the predictive ability of each model.

The SRTR is currently using a LASSO penalty, which prefers models with smaller absolute effect sizes that still fit the data well.  So, there are some predictors with a log hazard ratio of zero (hazard ratio = 1), which have no effect on the predicted risk, and other predictors with non-zero log hazard ratios (hazard ratios greater than or less than 1) which have an effect on the predicted risk.  The penalty itself is chosen through cross-validation in order to find the penalty that maximizes the ability of the model to predict outcomes.

If you are interested in "statistical significance" as a proxy for which predictors are "important", then the predictors with hazard ratios unequal to 1.0 are "important" since the model-fitting penalty would have set those hazard ratios to 1.0 if those predictors didn't improve the model predictions enough to overcome the penalty.

Are candidates listed for multi-organ transplants included in the transplant rate and waitlist mortality calculations for each organ separately?

Yes. If a candidate is on the waiting list for a particular organ type, he or she will be included in the separate cohorts. Until the transplant occurs, SRTR cannot be certain that a multi-organ transplant will occur.

Program Specific Reports (PSR) Methodology- Posttransplant Outcomes

Where are the cohorts for the various Posttansplant tables in the PSR listed?

We have them listed on the page of the SRTR public website here: PSR Reporting Timeline and they are updated every time the new PSRs and OSRs are publicly released in January and July.

Some of our patients in the cohort aren’t showing a death or graft failure.

Patients who undergo transplant in the last 6 months of the accrual period for the 1-year reporting time point are followed for only 6 months after transplant because the 1-year follow-up information is not yet available in the current OPTN data. Standard survival analysis methods are used to incorporate the first 6 months of experience for this subset of patients. (See diagram for more detail.)

How does SRTR define a multi-organ transplant?

Generally, SRTR classifies a transplant of two or more organs from one donor into a single recipient as a multi-organ transplant. Currently, heart-lung and kidney-pancreas transplants are the only multi-organ types included in the adjusted posttransplant outcome evaluations.  However, unadjusted posttransplant outcomes of multi-organ transplants are included in the program-specific reports.

Are recipients of multi-organ transplants included in the calculations for posttransplant survival?

Currently, all multi-organ transplants are excluded from the adjusted posttransplant survival calculations for kidney, heart, lung, and liver. Pancreas and intestine transplant outcomes are treated differently. Please see the Technical Methods documentation for details regarding pancreas, kidney-pancreas, heart-lung, and intestine transplant outcome inclusions and exclusions.

What is the historical rationale for excluding multi-organ transplants? 

In the past, multi-organ transplants were so rare that SRTR could not reliably estimate the risk associated with the procedure type. Today, the numbers are more substantial. Several types of multi-organ transplants are now relatively common, such as kidney-liver or heart-kidney.

Why are heart-lung reports released, but not liver-kidney reports? Heart-lung transplants occur less frequently than liver-kidney. Are any changes planned? 

Following recommendations from the OPTN Policy Oversight Committee, HRSA has encouraged SRTR to consider the inclusion of multi-organ transplants. After working with HRSA and our SRTR Visiting Committee, SRTR has begun to implement methods for including multi-organ transplants in models. However, it will be some years before all models are rebuilt with multi-organ inclusion.

How are retransplants handled when assessing outcomes in the program-specific reports?

Retransplants are included in the analysis of graft outcomes; however, they are excluded from analyses of patient survival. When assessing patient survival, only first transplants are considered.

Is a kidney transplant after a kidney-pancreas transplant considered a retransplant when assessing outcomes in the program-specific reports?

Yes. A Kidney was previously transplanted (in the KP transplant) so the next Kidney alone transplant is considered a retransplant.

If a candidate underwent transplant 10 years ago, are all subsequent transplants considered retransplants or only those that occur within a certain time period?

If SRTR has data to indicate that a patient previously underwent a particular type of transplant, any new transplant of the same organ type will be considered a retransplant. For two known reasons, SRTR may not have the data to indicate a previous transplant. First, if the transplant occurred before 1988, it will not be found in the OPTN database and SRTR will not have a record of it. Additionally, SRTR does not have patient-level data on all transplants performed outside of the United States. If SRTR does not have any record of a previous transplant, the new transplant would be considered the first.

Why Does OPTN include Retransplant as a diagnosis on the TIEDI forms, but SRTR doesn’t consider it a true diagnosis in their risk adjustment models?

For the purposes of risk adjustment, SRTR wants to look at what really contributed to a patient needing a transplant. Retransplant is not the reason the patient needed a transplant, rather the condition that lead to the need for a new transplant or the original transplant. In the case where retransplant is indicated as a primary diagnosis on the Transplant Recipient Form, SRTR will look at the primary diagnosis from the Transplant Candidate Form, or the primary diagnosis from the initial transplant. 

Are bilateral or en bloc transplants, counted as 1 or 2?

For the sake of graft survival statistics, we are counting the grafts, so one bilateral lung transplant is 1 graft. If lungs are split to 2 separate recipients, then it’s 2 grafts. For kidneys; 2 kidneys placed (en bloc) at the same time is still one graft. As long as the organs came from the same donor and were grafted into one recipient, it’s considered 1 graft.

Offer Acceptance Tables

What constitutes an offer? Does an offer have to be a primary offer for a recipient at a transplant center, or does an offer count for those for patients who are further down the match run?

Offers are only counted if they were accepted or were declined before an accepted offer.  This means that offers for eventually discarded organs are not included.

For the “hard-to-place” organs, does “over 50 offers” mean that there were more than 50 offers total, or that the organ was turned down over 50 times before it was accepted?

The data evaluates offers that were previously turned down 50 times or more.  This can include turn-downs at the same program.  Note that if an organ takes more than 50 offers to place, the first 50 offers are not included in the calculation of the offer acceptance ratio of “hard-to-place” organs.

Is an "acceptance" only counted if an organ is transplanted?

Yes, only transplanted organs are counted as accepted.

When a patient is status I or has a MELD score of greater than 35, and a liver is accepted for transplant, the center will continue to receive offers for that recipient as the recipient is not removed from the list until after the transplant. These offers have to be declined. Are these also counted as declines?

Yes, we include these offers because programs may accept and transplant these offers (leading to a late decline of the previously accepted liver).

Is organ offer acceptance data only for single organs?

Yes, offers to candidates listed for multiple organs at the program are not evaluated.

Are Multi-organ transplant excluded from these numbers?

Yes, offers to candidates listed for multiple organs at the program are not evaluated.

Risk Adjustment Models

What is risk adjustment?

One of the key functions of the SRTR is to assess transplant program performance. We currently do this by assessing how often patient transplanted at a program experience failure of their transplanted organ or die following their transplant. SRTR could simply report how many patients died or experienced graft failures, for example:

Program A: 10 patient deaths

Program B: 10 patient deaths

However, programs that transplanted more patients would be expected to have more deaths or graft failures, so this does not tell us anything about the quality of the program. Alternatively, we could simply present the percentage of patients that died or experienced graft failure. This is better because it takes into account, or “adjusts” for the size of the program. For example:

Program A: 10% of patients died (10 out of 100 transplanted patients)

Program B: 5% of patients died (10 out of 200 transplanted patients).

In the above example, program A and B both had 10 deaths, but program B transplanted twice as many patients, so their percentage of deaths was half that of program A.

Taking this example a bit further, we must recognize that not all patients transplanted at two different programs are alike... some programs may transplant more patients with more complicated illnesses or older patients. A program taking on more complicated cases would be expected to have higher rates of death, even if their care teams were as capable as a program transplanting lower risk patients.

The SRTR attempts to adjust for the risk level of the transplants by taking into account many characteristics of the transplant recipients and the donors. By doing so, we are attempting to make a more “apples to apples” comparison of program performance. We do this using a series of statistical models that take many recipient and donor characteristics into account. These models give a prediction of how likely each transplant recipient was to die or experience a transplant failure if the program was performing consistent with the national experience for similar patients receiving similar donors. This allows us to compare the observed number of deaths or transplant failures at a program to an expected number of deaths or graft failures based on national experience. Complete details of the risk adjustment models and what factors are included in these models are available here: Posttransplant Risk Adjustment Models.

I’m unable to see the risk adjustment models. The screen I see is blank (or grey).

This is due to Internet Explorer degradation. The interactive risk adjustments are optimal in Chrome as a web browser. Firefox also works. After trying that, if it still doesn’t work right contact srtr@srtr.org for additional assistance.

Primary Diagnosis value is missing, although we entered “retransplant” on the TIEDI.

For the purposes of risk adjustment, SRTR wants to look at what really contributed to a patient needing a transplant. Retransplant is not the reason the patient needed a transplant, rather the condition that lead to the need for a new transplant or the original transplant. In the case where retransplant is indicated as a primary diagnosis on the Transplant Recipient Form, SRTR will look at the primary diagnosis from the Transplant Candidate Form, or the primary diagnosis from the initial transplant. If no diagnosis is found, the value will remain “missing”.

You used to have a document that explained where the covariates come from on the TIEDI forms and I can’t find it now?

If you go to the risk adjustment models page, here: Posttransplant Risk Adjustment Models the first view that comes up is the list of covariates or elements in the model. In general, elements labeled “donor” come from donor registrations, “recipient” from recipient registrations and “candidate” from candidate registration except where the candidate elements also have “last” in the description, then it comes from waitlist. Then the value is whatever the last measure taken before transplant.

How is a “missing” value treated in the risk adjustments and why?

A missing value is given the same risk as the lowest risk covariate in the element. This is because centers should be making sure they enter their data completely and accurately. Giving the missing values the lowest value encourages centers to ensure they are entering a value, rather than leaving it blank to possibly get a better risk adjustment.

There are elements included in data sheets, but not in elements list in the risk adjustment models online?

If you look at the models found here: Posttransplant Risk Adjustment Models there is a tab “Other Elements Considered.” It is probably listed there. All elements in the Data Integrity Reports are the elements LASSO chose as determinants of the risk; however, if the element did not return a value that contributed to this iteration of the model, it was taken out of the “Model Coefficients” list. This was simply to avoid confusion when people looked at the models and saw a series of “0” as the coefficient values.

Why do some adjustments seem counter-intuitive; previous malignancy, for example?

The models are fit to the available data.  If the available data suggests that a predictor is protective, the model will have a negative coefficient (it reduces the rate of graft failures or deaths).  Now, just because a predictor is protective doesn't necessarily mean that it is "good" in a clinical sense.  The models don't tell us why a predictor is associated with lower risks of graft failure.  For example, a history of malignancies could be correlated with something else (that isn't in the model), which is really causing the risk difference.  Or, a history of malignancies is correlated with something that is in the model, but recipients with both malignancies and the other factor are lower risk than recipients who only have the other factor, for some reason.  

You shouldn’t look at the coefficients in a multiply-adjusted model individually and draw any particular conclusions about whether the effects make sense.  After accounting for all the other factors in the model, malignancy history (or something correlated with it) is either harmful or protective.

Where are risk adjustment models from previous reporting periods?

On the risk Adjustment model page, just beneath the "Risk Adustment Model Documentation..." Title at the top of the page, you can select which reporting period you want to view. The system gives you the last period's models in the drop-down there. For older models, you will need to contact an SRTR representative at srtr@srtr.org.

What is LS?

It is a Linear Spline. A linear spline plots a curve on a chart. The Right/Left LS means, respectively, a right or left linear spline. Using “age” as an example: ‘Left LS at 50 years old’ is the linear effect for candidates starting at 50, e.g., a 35 year old candidate would have a value of 50 – 35= 15 (this value would be 0 for all candidates over 50 years old at listing).  Conversely, ‘Right LR at 30 years old’ is the linear effect for candidates starting at 30, e.g., a 35 year old candidate would have a value of 35 – 30 = 5 (this value would be 0 for all candidates under 30 years old at listing).

When are the risk-adjustment models rebuilt? 

SRTR has implemented a 3-year rolling cycle to completely rebuild each organ’s risk adjustment models. The cycle started with kidney models in 2014 and heart and lung models were rebuilt in 2015. Liver was rebuilt in 2017 and intestine models are anticipated to be rebuilt in 2018.

What is the difference between the Estimated and Expected probability?

The estimated probability is not risk adjusted estimates the probability of graft survival or patient survival at the program.  Because it is not risk-adjusted, it is not a metric of program quality. It is based on the Kaplan-Meier survival function estimate for the program. The expected probability is the average modeled probability of graft or patient survival based on the characteristics of the recipients and donors at the program, but not on the actual outcomes at the program.  Since it is not based on program outcomes, it is also not a metric of program performance.  Instead, it is a metric of program risk tolerance.  Programs with higher risk recipients, on average, will have lower expected survival.

What is the difference between a “rebuild” and a “refit” of the risk adjustment models?

A refit is different from a rebuild in that the same elements are used, but the values indicated for the covariates may change depending on the data collected for that reporting period. A rebuild of a model entails completely re-evaluating the elements and their determinant of the actual level of risk. During the rebuild process, elements may be removed and others added. The models are “refit” each six-month cycle based on the data collected.

Why isn’t the estimated probability shown the same percentage I get when I divide the observed failures by the transplants observed?

The estimated probability is derived through the Kaplan Meier method (Cox 1972, Kaplan-Meier 1958), and that is why we use the term “estimated” instead of “observed”. Because follow-up was not complete for all transplant recipients through the end of the time interval, but all available follow-up data for each graft were used in the calculation of the statistics reported, the Kaplan-Meier formula adjusts for that.