Optimizing ML pipeline

[jankogasic]

Hi,

My pipeline setup:

1. query Spark remote servers
2. save data as libsvm file type to HDFS
3. fetch libsvm data to host memory with pandas.sql() (really long)
4. using sklearn.datasets.load_svmlight_file() to get CSR matrix (really long)
5. now use xgb.DMatrix(CSR_matrix) to finally start training

Is there a way to make this process faster? To skip some steps?

Potential ideas:

I would try to avoid pandas in step 3 and use webhdfs. Also, to try to avoid sklearn in step 4 and load libsvm file directly to xgb.DMatrix().

Is there a way to load data from Spark memory or HDFS directly to xgb.DMatrix() object? This way I can avoid hosts disk which might improve pipeline speed.

[trivialfis]

Not a Spark expert. But here are a few things that might be useful:

• Don't use text-based formats like libsvm or CSV. Use binary formats like Parquet or numpy format, etc.
• Use QuantileDMatix for the dtrain (training dataset) instead of DMatrix when using the hist tree method (default).
• Don't use DMatrix to load files, use pandas to load parquet or numpy to load its data, then pass them to XGBoost.
• If your machine has enough storage to store the data during training, you can fetch it from the remote server once and reuse it.

[trivialfis]

In addition to the above comment, you can consider using the (py)spark interface of XGBoost. However, getting rid of the text-based inputs should be the first step before trying anything else.