分类器之稀疏矩阵数据预处理

Question

背景：

训练集train.txt（674M）预测集test.txt（168M），每个数据项（向量）均为稀疏的。

格式为：

train.txt:

label index1:value1 index3:value3 index4:value4 ...

test.txt:

id index1:value1 index2:value2 index3:value3 ...

所预测的结果:

id,label

本人代码：

SetDirectory["D:\\college\\Junior\\数据挖掘\\Homework\\Homework2"];
records = 
  ReadList["train.txt", Word, RecordLists -> True, 
   WordSeparators -> {" "}];
maxcol = 201;
trainSet = 
  SparseArray[#[[1]] -> #[[2]] & /@ 
       Map[Read[StringToStream[#], Number] &, 
        StringSplit[Drop[records[[#]], 1], ":"], {2}], maxcol, 0] -> 
     ToExpression@records[[#]][[1]] & /@ 
   Range[Length@records];
Clear[records];
c = Classify[trainSet, PerformanceGoal -> "Quality"]
Clear[trainSet];
records = 
  ReadList["test.txt", Word, RecordLists -> True, 
   WordSeparators -> {" "}];
testX = SparseArray[#[[1]] -> #[[2]] & /@ 
      Map[Read[StringToStream[#], Number] &, 
       StringSplit[Drop[records[[#]], 1], ":"], {2}], maxcol, 0] & /@ 
   Range[Length@records];
Clear[records];
testY = c[#, "Probabilities"][1] & /@ testX;
Clear[testX];
Clear[c];
Ans = Join[{{"id", "label"}}, 
   Transpose@{Range[0, Length[testY] - 1], testY}];
Clear[testY];
Export["prediction.txt", Ans, "csv"]

本人代码写的丑，机器配置16G内存，无限卡在测试集稀疏矩阵的生成上，求大佬分析一波。。