Skip to content

Reading Error: All zeros when reading huge ply file (>2GB on Windows, >2.48GB on Ubuntu) #67

@zyqhnu

Description

@zyqhnu

All the points are (0, 0, 0) when I read a huge .ply file, both on Windows and Linux.
I seem to have figured out why:
I pre-load the entire file upfront (preload_into_memory = true), just like in example.cpp and example-utils.hpp. (It seems to be prepared for files smaller than 1GB. It's already stated in the notes.)
In my experiment, this occurs when the .ply file > 2.01GB (2160000275 bytes) on Windows, and > 2.48GB (2685605907 bytes) on Ubuntu.

But the memoryStream capacity is automatically expanded according to the reference

[@ SaiKishor-MSFT]
The maximum size of a MemoryStream object that can be handled by the System.IO.MemoryStream class is determined by the amount of available memory on the system. The maximum size of a MemoryStream object is 2 gigabytes (GB) by default.

My questions are:

  1. Do others get an all-zero error when reading a large file? (Does the capacity of MemoryStream automatically expand?)

  2. If all-zero errors are common, do you need to add a judgment: If the file size is too large, preload is not allowed? like this (in example.cpp, line 64):

// For most files < 1gb, pre-loading the entire file upfront and wrapping it into a 
// stream is a net win for parsing speed, about 40% faster. 
std::ifstream file_helper(filepath, std::ios::binary);
file_helper.seekg(0, std::ios::end);
std::streamsize byte_size = file_helper.tellg(); // get the byte_size of the file to be read
file_helper.close();
if (preload_into_memory && byte_size < 1.9e9)
{
    byte_buffer = read_file_binary(filepath);
    file_stream.reset(new memory_stream((char*)byte_buffer.data(), byte_buffer.size()));
}
else
{
    file_stream.reset(new std::ifstream(filepath, std::ios::binary));
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions