Repository navigation
PairTools sort failure on merged data #84
Description
Hello. I am trying to run PairTools on some mammalian HiC data. I have aligned the data with bwa and have the bam files ready. The first step (in which ${a} is the bwa alignment output bam file) finishes fine:
samtools view -h ${a} | pairtools parse -c ${Chromosomes} -o ${a}.pairfile
Then, I try the following step:
pairtools sort --nproc 8 -o ${a}.pairs.sorted ${a}.pairfile
Here, I receive an error that the program "cannot write compressed block". So, I added a flag to ensure the input is decompressed:
pairtools sort --cmd-in -o ${a}.pairs.sorted ${a}.pairfile
However, now the script no longer reads the ${a}.pairfile as input and reports that " Error: Got unexpected extra argument (Mutant-Merge.bam.pairfile)".
When I used the same script for the individual replicates before merging, everything worked fine. I merged them by simply concatenating the fastq files for the two replicates together, which I do not believe should affect this step, since the alignment ran fine.
Thank you for any feedback you may have.
Activity
[-]PairTools sort [/-][+]PairTools sort failure on merged data[/+]golobor commentedon May 29, 2020
Hi,
I apologize for the delayed response. Thank you for your interest in pairtools and your detailed report!! This looks puzzling indeed!
What operating system are you using for this example?
One issue I suspect is that you may not have write access to the Unix $TMPDIR. Could you please try running the same pairtools sort command with an extra argument --tmpdir=./ ?
Anton.
dowenlab commentedon Jun 1, 2020
Thank you so much for getting back to me! This example was using Windows 10 through an Ubuntu on Windows terminal. I have resubmitted the job including the tmpdir flag and will update to let you know how it goes!
UPDATE:
Looks like the tmpdir was the issue! The sort step successfully finished in ~3 days.
Here is the command that I ran successfully:
pairtools sort --tmpdir=./ --nproc 8 -o ${a}.pairs.sorted ${a}.pairfileconchoecia commentedon Jun 23, 2020
I am getting the same error with pairtools parse.
Error 36 : Write error : cannot write compressed block.The command is
pairtools sort --nproc 10 -o file_parsed_sorted.pairsam.gz file_parsed.pairsam.gz, pairtools version 0.3.0, on ubuntu 18.04 LTR.The command completed successfully when I added the
--tmpdir=./option. Maybe it would be good if this flag were a default, @golobor ?golobor commentedon Oct 27, 2020
@conchoecia , @dowenlab sorry for a half-year long reply :) This issue came up again, so I really need to fix it. :-(
The reason I hesitated with setting
--tmpdirto./by default is becausepairtools sortmay sometimes operate on a pipe, w/o explicit input or output files. In this case, the current folder can be any folder in the file system, which would make it unpredictable. One alternative could be to attempt using$TMPDIRand, if it's inaccessible, switch to./. But I would appreciate your experience in this issue. On the system where you got theError 36, was the/tmpfolder read-only or, instead, it had limited space that got overflowed bypairtools sort?..conchoecia commentedon Oct 27, 2020
Hi @golobor, in my case, and what I imagine to be the case for many people,
/tmpis just a small startup disk (only a few hundred GB) on our server. The working directory is a many-TB RAID disk, so plenty of space there. Looks like our home directories probably were full, and/tmpjust ran out of space.Good luck in whatever you decide!
golobor commentedon Oct 28, 2020
Hi @conchoecia , oh I see, thank you for the info, it's very helpful!!
I'd need to think a bit about a possible solution. There doesn't seem to be a one-size-fits-all solution - $TMPDIR is close to that, but doesn't work for a few people; using ./ by default could lead to really unpredictable behavior. A couple of things I could do is (a) to check the free space in /tmp and $TMPDIR and issue a warning if it's <10-20G; (b) improve the error message such that, if sort fails, suggest to use a different --tmpdir. Do you think that'd be useful?
conchoecia commentedon Oct 28, 2020
Hi @golobor - Just came back to make another case for using
./by default. Just look at what most other major bioinformatics tools do, genome assemblers,bwa/minimap/samtools, bedtools... most of these software use./to write tmp files. Why would your tool using./lead to unexpected behavior? Usually when people work in a directory, they do so because they know that there is space there and they intended to use it in the first place.golobor commentedon Oct 29, 2020
agalitsyna commentedon Apr 8, 2022
Thank you for the nice discussion. There is a problem with local temporary directory that I personally don't like, it's temporary files that have to be cleaned up manually if the command is forcefully terminated. If you run multiple tests, this might lead to accumulation of temporary files, which can be easily forgotten. Other ideas/thoughts? @Phlya or others?