Compare patient references

[sjspielman]

Purpose/implementation Section

Towards #1424

(edit: reminder that I won't be able to get back to this until at least week of 8/17, so review timeline can match!)

What is the goal of this pull request?

This PR adds a notebook where I ran SingleR over a set of samples using several reference strategies, focusing on patient samples and references.

Samples include as control a lung met sample, a bone primary sample, and then the 5 additional patient met samples which do not appear in lung.
References include metastasis, primary, both provided to SingleR in the same call, or a merged reference where I concatenate them (both ensure labels are identifiable eg T cells from both aren't collapsed) and send that single reference into SinglerR.

To evaluate results, I have barplots of the cell types they infer as well as heatmaps comparing different reference results. For all plots, I focus on just the top 10 cell types returned per reference so it doesn't get too out of control. The heatmaps also include the number of inferred cell types in the labels to provide additional context.

We would hope for a few things here:

• tight correspondence between the two combined reference approaches - we sometimes see this, but sometimes there are pretty salient differences
• tight correspondence for controls:
  • for a primary bone, I would hope that results from the primary reference are similar to a combined reference, and for a lung metastasis I would hope that results from the mets reference are similar to a combined reference. Admittedly I have only 2 control samples here so it's not a full exploration, but we only see this really for the primary, not the metastasis whose combined refs are coming back with a blend of cell types from the references

Although there is a decent amount of noise in many of these results, we do see some clear cell types consistently emerging across reference strategies. The noise makes it hard to nail down a precise strategy, so I tend to lean towards just using a combined reference, and as I sort of start to get at in the notebook I think this might be the concatenated one (but I remain back and forth here).

There are other analyses I did not include here that would be good to explore, such as looking at delta next or the SingleR result heatmaps themselves. I didn't want to include them just yet since there's already quite a bit here that I thought would be good to get more eyes on before expanding further in case you see patterns that I am missing, as my eyes now need a break from looking at heatmaps!

The renv environment has been updated as well. Here's the notebook:
03-compare-patient-references.nb.html

Note that it's probably true some text in the notebook needs to be fleshed out further, but again I wanted this to go through a round of review before polishing. The conclusions also need to be nailed down and I think we'll benefit from some discussion before doing that.

Author checklists

Analysis module and review

• This analysis module uses the analysis template and has the expected directory structure.
• The analysis module README.md has been updated to reflect code changes in this pull request.
• The analytical code is documented and contains comments.
• Any results and/or plots this code produces have been added to your S3 bucket for review.

Reproducibility checklist

• Code in this pull request has been added to the GitHub Action workflow that runs this module.
• The dependencies required to run the code in this pull request have been added to the analysis module Dockerfile.
• If applicable, the dependencies required to run the code in this pull request have been added to the analysis module conda environment.yml file.
• If applicable, R package dependencies required to run the code in this pull request have been added to the analysis module renv.lock file.

[allyhawkins]

Thanks for getting this comparison started and there is definitely a lot to think about. I have a lot of ideas for things that we could be explored, but I also want to make sure we keep this PR well-scoped. So I think we can use a lot of what you have here, but I think we need to take a step back before we get into the "tricky" samples.

To me, the main question that we are trying to answer is: is it appropriate to use both primary and metastatic references together or should we treat them separately based on disease timing? I think to answer the question about which reference is "appropriate", we need some sort of ground truth or as close to a ground truth as we can get. Because of that, I would make some adjustments to the experimental design.

I would pick two "control" samples, one sample that's part of both ScPCA and OsteoCar for primary and one for metastasis. Then for each sample you run primary or mets only and compare to a joined reference. For each sample/reference combination, you can compare the labels to the original labels and quantify how many cells get the same label. I want to see how much this number changes when you use primary-only vs the combined versions.

Along that same line, I would add a harmonized version of the reference where you don't include "prim" or "met" in the label so that CAFs from both get the same label. Then compare those results to the both and merged versions, where you do have a label. Do you see that cells are evenly split between the primary and mets version of the same label or do they favor one over the other?

I think once we narrow in on an approach that would work well for the majority of samples then we can revisit the "tricky" samples.

[allyhawkins]

I don't know if this is where I would start the exploration. I think I first want to know how do these references perform in samples that are actually part of OsteoCAR vs other samples with similar characteristics that are not part of OsteoCAR. We don't really have a good ground truth for these samples that have a mismatch, so I think we first need a baseline of how each reference performs on samples where we do have some sort of expectation or "ground truth" to compare to.

[allyhawkins]

This sentence is confusing to me and I had to go back to the PR comment to understand what you meant by controls. I would just add more context about what you mean by control here.

[allyhawkins]

I think the key comparison is missing here, which is comparing the tissue-appropriate reference (eg.., primary for primary samples and metastasis for metastasis samples) to the merged. And then the same comparison to the "both" reference. I want to know if you use the combined reference compared to the single reference, are you gaining any additional cell types that might not be present in one of the references? And if you are, do those cell types seem like valid matches? This is where having the delta next information would be helpful, but only for those "new" cell types.

Additionally, do you see that cells that get assigned to celltype X in the single reference, do those cells get split across "prim_celltypeX" and "mets_celltypeX" or are they consistently assigned to the appropriate reference type.

[allyhawkins]

To me this is the more informative of the plots.

[allyhawkins]

This tells me we probably need to use a combined reference, but I also think this is where delta next could maybe be helpful. In particular, for the cells that match to a primary cell type in the reference instead of a mets cell type, how confident are we in that prediction and what's the second best label?

For example, if the cell type is "mets_Fibroblasts" when using mets only but those cells get split between primary fibroblasts and mets fibroblasts then could we instead harmonize the labels?

[allyhawkins]

Just a small organizational note that this would be nicer to have below each section of plots. Its hard to scroll up and down for the conclusions and the plots.

[sjspielman]

@allyhawkins sending an update - I've been working on the this and the overhaul is going to be pretty extensive, even though some of the code is shared. It's going to make a lot more sense for a fresh PR, so I'm going to go ahead and close this. Also note, I agree that delta.next is important to bring in here, so I have decided to drop the approach where we literally provide both references to SingleR, because this does not return a delta.next due to the more involved procedure.

I'll also point out, because of SingleR's scores fine-tuning step, my understanding is that it is not definitively possible to determine the actual "second best label." For example, there are many times that the second best label based ont he returned scores alone is in fact that is reported as the actual assigned label. But, I think the SingleR heatmaps get us close to what we want to actually be answering here so that is a strategy I am using.