Compare reference use on patient samples from OsteoCAR by sjspielman · Pull Request #1428 · AlexsLemonade/OpenScPCA-analysis

sjspielman · 2026-08-20T17:50:25Z

Purpose/implementation Section

Please link to the GitHub issue that this pull request addresses.

Part of #1424
Emerged out of the closed PR #1427

What is the goal of this pull request?

The goal of this PR is to do some reference exploration specifically for the patient references at the Ann_Level2 annotation.

I chose two samples with good barcode overlap between ScPCA and OsteoCAR, one primary and one metastasis (but I do not show my work here! There is another exploratory notebook I can send in as a separate PR to see precisely how I did this, please let me know if this is something you'd want to review. I didn't think it was critical to record so I didn't want to waste reviewer time on it, is why it's not here.).
I run with three references: the one matched to its type (prim/mets), a harmonized version (same labels collapsed into one), and a concatenated version (labels retain their original prim/mets identity). I do not use a strategy where I provide both to SingleR since this nixes delta.next scores and these are pretty informative imo.
I then compare cell types in a variety of ways:
- Barplots of the cell types identified per reference, along with the delta.next distributions for those cell types.
- SingleR heatmaps which give us a sense of what other cell types are contending for the label
- Jaccard heatmaps comparing the labels to ground truth annotations
- Note that at one point I considered some actual accuracy measures e.g. with caret, but since the matrix comparison isn't square this is a bit tricky to interpret since some of the assumptions of confusion matrix metrics are violated. The plots seemed to communicate well enough what was going on without putting a number on it I think; let me know if you disagree.
The analysis is the same between the primary and the metastasis sample, but with conclusions interspersed throughout the text.

There are also quite a few functions used here, so I split them out to be called as a child report since it's just way too overwhelming otherwise, and the code is long enough as is!

One of the conclusions I draw here is we might want to do this same exploration with Ann_Level1 next, since the "divvying up" of those cell type seems to create challenging noise. But, the harmonized reference seems like a reasonable bet otherwise.

03-compare-patient-references-osteocar.nb.html

Provide directions for reviewers

What are the software and computational requirements needed to be able to run the code in this PR?

Laptop, renv

Are there particularly areas you'd like reviewers to have a close look at?

The notebook!

Is there anything that you want to discuss further?

We'll discuss how many next steps I should take, if any, after this PR. For example, I could pursue analogous notebooks for the xenograft references or for the patient reference but at Ann_Level1.

Author checklists

Analysis module and review

This analysis module uses the analysis template and has the expected directory structure.
The analysis module README.md has been updated to reflect code changes in this pull request.
The analytical code is documented and contains comments.
Any results and/or plots this code produces have been added to your S3 bucket for review.

Reproducibility checklist

Code in this pull request has been added to the GitHub Action workflow that runs this module.
The dependencies required to run the code in this pull request have been added to the analysis module Dockerfile.
If applicable, the dependencies required to run the code in this pull request have been added to the analysis module conda environment.yml file.
If applicable, R package dependencies required to run the code in this pull request have been added to the analysis module renv.lock file.

allyhawkins

This looks like a good start and I think will definitely be helpful in determining the reference type we use. I had one suggestion for an additional plot that I think should be added to better compare delta next values across reference types. Other than that I just left some general food for thought questions and comments.

allyhawkins · 2026-08-26T18:12:42Z

+
+
+We'll define the samples we're using here.
+These were chosen based on manual inspection of how many cells are present and how many barcodes overlap with `OsteoCAR` to be able to get some decent signal here.


Just curious what was the percentage of overlap for each sample? That might be helpful to know and record here.

Just curious what was the percentage of overlap for each sample

To confirm, do you mean just these or all of them? If all samples, I'd put that into a different notebook.

I think having the information for just these samples is sufficient.

allyhawkins · 2026-08-26T18:15:34Z

+In other words, if both references have T cells, we end up with two labels in `concatenated_aggr`, e.g. `mets_tcell` and `prim_tcell`
+
+Note that the `harmonized_aggr` and `concatenated_aggr` are created from `prim_aggr` and `mets_aggr`, because of memory.
+In other words, the raw references could not be combined on their own.


What do you mean that the raw references couldn't be combined on their own?

If I try to cbind SCE objects before aggregating them, R always runs out of memory no matter how much I gave it. I was only able to combine references after I aggregated each individual reference. This means that the pseudobulking is done for each reference separately - the same cell type label will be pseudobulked just for metastasis cells and separately just for primary cells, and those pseudobulked outputs end up getting combined with cbind.

Hmmm it might be worth creating a combined object and then downsampling each cell type to have X number of cells. That might ensure we are properly creating the combined references but then we are dealing with the effect of downsampling. I think if we decide to use a harmonized reference we should test both ways of building the reference to see if there are any differences.

Hmmm it might be worth creating a combined object and then downsampling each cell type to have X number of cells.

I can do this, but again it would be reversed - need to downsample cells in order to combine them in the first place.

I can do this, but again it would be reversed - need to downsample cells in order to combine them in the first place.

I think you should be able to cbind them, downsample, and then create the aggregated reference.

allyhawkins · 2026-08-26T18:20:31Z

+Next, we'll run `SingleR`:
+
+```{r}
+results <- run_singler(


I might be missing something, but it looks like this just produces one result from SingleR? I think you might be missing code that actually runs this on each reference and then creates a list of results to use as input to making the plots below.

This function in defined in the exploratory-utils.Rmd file: https://github.com/sjspielman/OpenScPCA-analysis/blob/5aa8dfa6a9fe9f9842be5ef7486cb98428cf009d/analyses/cell-type-osteosarcoma/exploratory-notebooks/exploratory-utils.Rmd#L13

It runs on three different references. I can add more comments to be clear where functions come from.

Can you actually move this function in particular to be in this notebook? I think it's easier to follow for exploratory notebooks to have some of the key implementations, in this case how SingleR was actually run in the notebook itself.

allyhawkins · 2026-08-26T18:26:16Z

+Across all references, we see various tumor cell types coming back as most common.
+When a primary reference is used or explicitly included as a standalone cell type, we see that proliferative tends to dominate, but the harmonized reference has basal progenitors dominating.
+These are the second highest category for primary and concatenated. 
+Labels for the concatenated reference tend to be primary, not metastasis.


I think it would be helpful to add a plot here that specifically focuses on one cell type and compares the delta next scores for those cell types across methods. For example, if you look at cells that are always labeled as proliferative cells, regardless of the method, how does the delta next score change? You can maybe get an idea of that here but with different x axes limits it's hard to make a direct comparison.

What I would envision is a sina or ridge plot with the reference type on one axis and then the other axis is delta. next. Only cells that are consistently labeled the same across all three types should be included to see how the "confidence" changes with each reference type.

allyhawkins · 2026-08-26T18:30:54Z

+
+There is certainly some good agreement, but we again the the same pattern play out there isn't always precise agreement between which type of tumor cell is bring called.
+This story is similar across all references used.
+This suggests we might actually be well served by using `Ann_Level1`, even though it is less fine-grained.  


I tend to agree with this. For the primary reference only we see that we are almost doubling the amount of cells that are getting labeled as proliferative. This tells me we may not have enough information in the SingleR model to properly differentiate the other tumor cell types with this level of annotation.

allyhawkins · 2026-08-26T18:36:59Z

+
+```{r}
+#| fig.height: 6
+SingleR::plotScoreHeatmap(results$`Concatenated reference`, main = "Concatenated reference")


One thing I note about this plot, which I think is really helpful to us deciding between harmonized vs concatenated is that there's a group of immune cells that appear to be "mets_T_cells" also have high scores for "prim_T_cells". You see a similar phenomenon with the proliferative tumor cell types. This is the information I'm looking for to help determine if specifying if the reference is from primary or mets is important or not, and I'm not convinced that it is.

allyhawkins · 2026-08-26T18:41:04Z

+  patchwork::wrap_plots(ncol = 1)
+```
+
+On the whole, this is a much cleaner result compared to the primary sample we explored in the previous section, but we also have many more cells being compared here!


I think you mean we have more cell types being compared here. I agree that this is cleaner, but this is probably because the mets sample has both normal and tumor cells and we don't see that for the primary samples. I wonder how representative this primary sample is to the rest of the primary samples? Are most of them just tumor cells or do we see evidence for non-tumor cells. I think if we have samples with a "ground truth" and both normal and tumor cells, it would be interesting to see how they compare to this primary sample. You don't need to do that here, but it would be helpful to know.

I think if we have samples with a "ground truth" and both normal and tumor cells, it would be interesting to see how they compare to this primary sample. You don't need to do that here, but it would be helpful to know.

This suggests me it would be helpful for me to send in a separate PR with a notebook looking over all ScPCA samples that are also in OsteoCAR, to overall look at the overlap of barcodes and the cell type distributions? Not to run SingleR on, but to get a sense of the landscape there.

Yes that could be informative to get some basic information on where we are starting with this dataset.

I wonder how representative this primary sample is to the rest of the primary samples? Are most of them just tumor cells or do we see evidence for non-tumor cells.

Let me just note: In exploring other samples one could use here, what I found is that samples with more diversity in cell types tend to have many fewer overlapping barcodes with the ScPCA version of the library, which makes the ground truth comparison harder to do simply because of dataset size.

allyhawkins · 2026-08-26T18:42:02Z

+We see (in particular for primary patient sample) that there tends to be ambiguity among the tumor cell types, as the `SingleR` heatmaps emphasize.
+One limitation here is that the primary sample doesn't have a ton of cell type diversity besides tumor cells.
+
+Based on this exploration, the harmonized reference (where we combine aggregated references and _merge the labels_) seems like a decent bet, but it would also be worthwhile to explore using the coarser-grained `Ann_Level1` to see what we would gain vs lose compared to `Ann_Level2`.


Before comparing what we gain or lose, we should just see if we can better recapitulate the ground truth with the less granular cell types.

…re-patient-references-osteocar.Rmd Co-authored-by: Ally Hawkins <54039191+allyhawkins@users.noreply.github.com>

…elete

… of github.com:sjspielman/OpenScPCA-analysis into sjspielman/compare-patient-references-osteocar-samples

sjspielman · 2026-08-27T13:26:05Z

I want to note that we discussed offline taking the following next steps here:

Complete this notebook with the current references as implemented. Since prim/mets references as-is (in their original SCE form) cannot be readily combined locally due to memory limitations, we'd want to downsample these references first before concatenating the raw versions, for testing different approaches to make a combined reference. Exploring the strategy for combining references is something to pick up later.
Add a PR with a second notebook doing a similar exploration but using the Ann_Level1 reference instead. This may also involve using Ann_Level2 for non-tumor cell types but collapsing all tumor cell types into a single Tumor category, as Ann_Level1 reflects.

…k at the percent of overlap and cell types before running singler

…r plotting function for looking at delta.next of shared cell types

sjspielman · 2026-08-27T15:47:45Z

Based on the scope we have discussed for this PR, this should be ready for re-review.

I brought the run_singler() function into the main notebook, but left the rest in the utils Rmd.
At the beginning of each sample analysis section, I include the fraction of ScPCA cells that are present in OsteoCar as well as barplots of the "ground truth labels," faceted by ann_level1.
I added one function to be able to make the plot you requested, which shows delta.next scores across references for matching labels. I ended up making this as ridge plots faceted by cell type so we can have a close comparison. There is some minor signal here, but there is no definitive trend; distributions are generally the same.

Let me know any other changes or recommended additions! After this, I'll proceed to do a fairly similar notebook but with the less fine-grained tumor annotations.

03-compare-patient-references-osteocar.nb.html

allyhawkins

LGTM. I just have one small comment about turning the cell type plot into a table but either one is fine.

allyhawkins · 2026-08-28T15:33:57Z

+What are the cell types as recorded in OsteoCAR?
+
+```{r}
+ggplot(prim_ground_truth) +


I think this plot is fine, but you could also just print out a table of cell type counts. I think that would make it easier to see the imbalance between tumor and normal cells.

I ended up making it facet_grid() which I thinks get the point across better! This is easier for me to read than a table which spans multiple pages, so this way we see it all at once.

allyhawkins · 2026-08-28T15:36:13Z

+#| fig.height: 5
+#| message: false
+
+plot_delta_next_per_reference(results)


This plot is helpful thank you! I personally don't see any clear trend in one reference always being better than the others, which is good information to know. That tells me we don't get any benefit from using the primary only vs a combined reference.

sjspielman added 6 commits August 19, 2026 09:37

txt extension wrongly committed, make it tsv

13a11c0

utils functions in a standalone file

8b946d6

snapshot

263fccd

add to readme

f96aa58

the actual notebook

965a492

rename

41fff96

sjspielman requested a review from jaclyn-taroni as a code owner August 20, 2026 17:50

sjspielman requested review from allyhawkins and removed request for jaclyn-taroni August 20, 2026 17:50

typos

5aa8dfa

allyhawkins reviewed Aug 26, 2026

View reviewed changes

sjspielman and others added 4 commits August 26, 2026 15:12

Update analyses/cell-type-osteosarcoma/exploratory-notebooks/03-compa…

69e4db3

…re-patient-references-osteocar.Rmd Co-authored-by: Ally Hawkins <54039191+allyhawkins@users.noreply.github.com>

Update analyses/cell-type-osteosarcoma/exploratory-notebooks/03-compa…

9d5a1f4

…re-patient-references-osteocar.Rmd Co-authored-by: Ally Hawkins <54039191+allyhawkins@users.noreply.github.com>

a few more docs for functions, and remove a function i had meant to d…

d0065ed

…elete

Merge branch 'sjspielman/compare-patient-references-osteocar-samples'…

c562912

… of github.com:sjspielman/OpenScPCA-analysis into sjspielman/compare-patient-references-osteocar-samples

sjspielman added 5 commits August 27, 2026 09:49

move the singler function into the main notebook, and add code to loo…

da4993f

…k at the percent of overlap and cell types before running singler

update get_top_n to be more flexible for input column, and add anothe…

e4ccb98

…r plotting function for looking at delta.next of shared cell types

comment

97c7136

add plots and associated text using the new function

fa857ee

knit

228a4cd

sjspielman requested a review from allyhawkins August 27, 2026 15:47

allyhawkins approved these changes Aug 28, 2026

View reviewed changes

switch to facet_grid so we can clearly see the different axis limits

cdbf551

sjspielman merged commit 2db91da into AlexsLemonade:main Aug 28, 2026
2 checks passed

sjspielman deleted the sjspielman/compare-patient-references-osteocar-samples branch August 28, 2026 17:44

This was referenced Aug 28, 2026

Compare patient references with Ann_Level1 for tumor/mesenchymal #1429

Merged

Evaluate reference approach for annotating osteosarcoma samples #1424

Open



		We'll define the samples we're using here.
		These were chosen based on manual inspection of how many cells are present and how many barcodes overlap with `OsteoCAR` to be able to get some decent signal here.

Uh oh!

Conversation

sjspielman commented Aug 20, 2026

Purpose/implementation Section

Please link to the GitHub issue that this pull request addresses.

What is the goal of this pull request?

Provide directions for reviewers

What are the software and computational requirements needed to be able to run the code in this PR?

Are there particularly areas you'd like reviewers to have a close look at?

Is there anything that you want to discuss further?

Author checklists

Analysis module and review

Reproducibility checklist

Uh oh!

allyhawkins left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sjspielman Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sjspielman commented Aug 27, 2026

Uh oh!

sjspielman commented Aug 27, 2026

Uh oh!

allyhawkins left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

sjspielman Aug 26, 2026 •

edited

Loading