背景：
训练集train.txt（674M）预测集test.txt（168M），每个数据项（向量）均为稀疏的。
格式为：
train.txt:
label index1:value1 index3:value3 index4:value4 ...
test.txt:
id index1:value1 index2:value2 index3:value3 ...
所预测的结果:
id,label
本人代码：
SetDirectory["D:\\college\\Junior\\数据挖掘\\Homework\\Homework2"];
records =
ReadList["train.txt", Word, RecordLists -> True,
WordSeparators -> {" "}];
maxcol = 201;
trainSet =
SparseArray[#[[1]] -> #[[2]] & /@
Map[Read[StringToStream[#], Number] &,
StringSplit[Drop[records[[#]], 1], ":"], {2}], maxcol, 0] ->
ToExpression@records[[#]][[1]] & /@
Range[Length@records];
Clear[records];
c = Classify[trainSet, PerformanceGoal -> "Quality"]
Clear[trainSet];
records =
ReadList["test.txt", Word, RecordLists -> True,
WordSeparators -> {" "}];
testX = SparseArray[#[[1]] -> #[[2]] & /@
Map[Read[StringToStream[#], Number] &,
StringSplit[Drop[records[[#]], 1], ":"], {2}], maxcol, 0] & /@
Range[Length@records];
Clear[records];
testY = c[#, "Probabilities"][1] & /@ testX;
Clear[testX];
Clear[c];
Ans = Join[{{"id", "label"}},
Transpose@{Range[0, Length[testY] - 1], testY}];
Clear[testY];
Export["prediction.txt", Ans, "csv"]
本人代码写的丑，机器配置16G内存，无限卡在测试集稀疏矩阵的生成上，求大佬分析一波。。
附train.txt与test.txt部分数据（上述奇丑无比的代码可以在小数据上跑过）
train.txt:
1 15:1 47:2 92:1.0 93:1.0 94:1.0 95:1.0 96:1.0 97:1.0 98:1.0 99:0.14285714285714285 100:0.07142857142857142 101:0.03571428571428571 102:0.020833333333333332 103:4.0 119:2.0 120:2.0 121:2.0 122:2.0 123:1.0 124:1.0 125:1.0 133:1 134:1 135:1.0 136:1 137:1.0 138:1 139:1.0 168:1
1 14:4 15:2 92:2.0 93:6.0 94:6.0 95:6.0 96:0.3333333333333333 97:1.0 98:1.0 99:0.2857142857142857 100:0.42857142857142855 101:0.21428571428571427 102:0.125 103:5.0 104:630.0 106:27506.2 107:2768967347.76 108:27506.2 109:2768967347.76 110:27506.2 111:2768967347.76
0 7:13 8:17 9:26 10:11 11:9 12:16 13:5 14:10 15:7 16:4 92:13.0 93:31.0 94:108.0 95:118.0 96:0.11016949152542373 97:0.2627118644067797 98:0.9152542372881356 99:1.8571428571428572 100:2.2142857142857144 101:3.857142857142857 102:2.4583333333333335 104:41141.5 105:1156292144.4166667 106:38132.73333333333 107:1840492430.795555 108:21983.6261682243 109:1040963841.9723995 110:21486.69230769231 111:979900569.9053252
0 1:3 2:5 3:3 4:1 6:1 7:6 8:3 9:2 10:2 11:1 12:3 13:3 14:2 15:1 16:1 92:2.0 93:10.0 94:22.0 95:37.0 96:0.05405405405405406 97:0.2702702702702703 98:0.5945945945945946 99:0.2857142857142857 100:0.7142857142857143 101:0.7857142857142857 102:0.7708333333333334 104:329587.0 106:125685.66666666667 107:16384229093.555555 108:113891.04761904762 109:13851726529.283443 110:113185.72222222222 111:14166796132.089508
0 1:40 2:59 3:42 4:33 5:37 6:39 7:39 8:47 9:37 10:38 11:66 12:40 13:19 14:16 15:25 16:33 50:1 92:61.0 93:114.0 94:330.0 95:610.0 96:0.1 97:0.18688524590163935 98:0.5409836065573771 99:8.714285714285714 100:8.142857142857142 101:11.785714285714286 102:12.708333333333334 104:9408.933333333332 105:169217349.12888888 106:10313.044247787611 107:183375153.79450235 108:7303.65349544073 109:125717366.16564879 110:6783.001642036124 111:117702208.87191832 129:1.0
1 11:3 16:1 32:6 43:2 59:1 64:17 92:1.0 93:1.0 94:4.0 95:4.0 96:0.25 97:0.25 98:1.0 99:0.14285714285714285 100:0.07142857142857142 101:0.14285714285714285 102:0.08333333333333333 103:1.0 108:391338.6666666667 109:306262945908.22217 110:391338.6666666667 111:306262945908.22217 112:6.0 113:6.0 114:6.0 115:6.0 116:1.0 117:1.0 118:1.0 121:2.0 122:2.0 125:1.0 126:17.0 127:17.0 128:18.0 129:18.0 130:0.9444444444444444 131:0.9444444444444444 132:1.0 133:1 134:1 135:1.0 136:1 137:1.0 138:1 139:1.0 169:1
0 21:2 31:2 32:9 103:48.0 112:11.0 113:11.0 114:11.0 115:13.0 116:0.8461538461538461 117:0.8461538461538461 118:0.8461538461538461
0 3:2 4:5 5:8 6:28 7:46 8:26 9:5 10:7 11:3 12:11 13:58 14:15 15:4 16:12 50:1 51:5 54:5 55:2 56:6 59:5 60:2 62:2 92:18.0 93:99.0 94:150.0 95:230.0 96:0.0782608695652174 97:0.43043478260869567 98:0.6521739130434783 99:2.5714285714285716 100:7.071428571428571 101:5.357142857142857 102:4.791666666666667 104:31118.470588235294 105:2812065466.719723 106:12004.479591836734 107:816436295.4536653 108:16041.03355704698 109:1312746287.6565914 110:15501.126637554586 111:1360229428.4599447 127:4.0 128:15.0 129:28.0 131:0.14285714285714285 132:0.5357142857142857
1 11:1 12:9 13:5 14:3 60:4 61:3 63:1 76:323 77:218 78:430 82:930 83:971 84:971 86:0.957775489186 87:1.0 89:66.4285714286 90:34.6785714286 91:20.2291666667 93:14.0 94:18.0 95:18.0 97:0.7777777777777778 98:1.0 100:1.0 101:0.6428571428571429 102:0.375 103:7.0 106:51483.11111111111 107:1349514220.9876544 108:55837.25 109:1793620241.5208333 110:55837.25 111:1793620241.5208333 126:1.0 127:8.0 128:8.0 129:8.0 130:0.125 131:1.0 132:1.0
0 1:5 14:20 15:24 16:66 30:2 46:4 48:1 62:3 64:1 65:321256 78:14027 79:76126 80:78756 81:168909 82:168909 83:168909 84:490165 85:0.3445962074 86:0.3445962074 87:0.3445962074 88:24129.8571429 89:12064.9285714 90:6032.46428571 91:10211.7708333 92:107.0 93:110.0 94:110.0 95:115.0 96:0.9304347826086956 97:0.9565217391304348 98:0.9565217391304348 99:15.285714285714286 100:7.857142857142857 101:3.9285714285714284 102:2.3958333333333335 104:8513.507246376812 105:140737891.14849815 106:8381.291666666666 107:137652471.09548613 108:8381.291666666666 109:137652471.09548613 110:55111.746666666666 111:157036874795.14914 112:2.0 113:2.0 114:2.0 115:2.0 116:1.0 117:1.0 118:1.0 119:3.0 120:5.0 121:5.0 122:5.0 123:0.6 124:1.0 125:1.0 126:1.0 127:4.0 128:4.0 129:4.0 130:0.25 131:1.0 132:1.0 133:1 134:1 135:1.0 136:1 137:1.0 138:1 139:1.0 140:1 141:1 142:1.0 143:1 144:1.0 145:1 146:1.0 147:1 148:1 149:1.0 150:1 151:1.0 152:1 153:1.0 167:1 185:1 201:1
1 1:1 2:2 15:3 92:3.0 93:3.0 94:3.0 95:6.0 96:0.5 97:0.5 98:0.5 99:0.42857142857142855 100:0.21428571428571427 101:0.10714285714285714 102:0.125 103:4.0 104:464.5 105:6642.25 106:464.5 107:6642.25 108:464.5 109:6642.25 110:712653.8 111:1748875371050.1602
test.txt
0 15:1 92:1.0 93:1.0 94:1.0 95:1.0 96:1.0 97:1.0 98:1.0 99:0.14285714285714285 100:0.07142857142857142 101:0.03571428571428571 102:0.020833333333333332 103:4.0
1 1:199 2:148 3:81 4:247 5:73 6:161 7:173 8:86 10:313 11:411 12:276 13:164 14:230 15:67 54:1 92:115.0 93:598.0 94:1608.0 95:2629.0 96:0.043742868010650436 97:0.22746291365538227 98:0.6116394066184861 99:16.428571428571427 100:42.714285714285715 101:57.42857142857143 102:54.770833333333336 103:4.0 104:2071.315789473684 105:45023368.65466296 106:1406.8442211055276 107:33892183.119786546 108:1273.0647168637213 109:138662884.3741557 110:1437.8264840182649 111:96328238.86182517 129:1.0
2 16:2 48:1 92:2.0 93:2.0 94:2.0 95:2.0 96:1.0 97:1.0 98:1.0 99:0.2857142857142857 100:0.14285714285714285 101:0.07142857142857142 102:0.041666666666666664 103:2.0 104:32664.0 106:32664.0 108:32664.0 110:32664.0 119:1.0 120:1.0 121:1.0 122:1.0 123:1.0 124:1.0 125:1.0 133:1 134:1 135:1.0 136:1 137:1.0 138:1 139:1.0 169:1
3 16:18 48:1 64:1 92:18.0 93:18.0 94:18.0 95:18.0 96:1.0 97:1.0 98:1.0 99:2.5714285714285716 100:1.2857142857142858 101:0.6428571428571429 102:0.375 104:10763.64705882353 105:469519587.5224914 106:10763.64705882353 107:469519587.5224914 108:10763.64705882353 109:469519587.5224914 110:10763.64705882353 111:469519587.5224914 119:1.0 120:1.0 121:1.0 122:1.0 123:1.0 124:1.0 125:1.0 126:1.0 127:1.0 128:1.0 129:1.0 130:1.0 131:1.0 132:1.0 133:1 134:1 135:1.0 136:1 137:1.0 138:1 139:1.0 140:1 141:1 142:1.0 143:1 144:1.0 145:1 146:1.0 147:1 148:1 149:1.0 150:1 151:1.0 152:1 153:1.0 169:1 185:1 201:1
4 15:6 47:5 79:116 81:116 82:116 83:116 84:116 85:1.0 86:1.0 87:1.0 88:16.5714285714 89:8.28571428571 90:4.14285714286 91:2.41666666667 92:6.0 93:6.0 94:6.0 95:6.0 96:1.0 97:1.0 98:1.0 99:0.8571428571428571 100:0.42857142857142855 101:0.21428571428571427 102:0.125 103:4.0 104:363.75 105:132944.1875 106:363.75 107:132944.1875 108:363.75 109:132944.1875 110:363.75 111:132944.1875 119:5.0 120:5.0 121:5.0 122:5.0 123:1.0 124:1.0 125:1.0 133:1 134:1 135:1.0 136:1 137:1.0 138:1 139:1.0 168:1
5 16:1 92:1.0 93:1.0 94:1.0 95:1.0 96:1.0 97:1.0 98:1.0 99:0.14285714285714285 100:0.07142857142857142 101:0.03571428571428571 102:0.020833333333333332 133:1 134:1 135:1.0 136:1 137:1.0 138:1 139:1.0 169:1
6 9:3 10:2 16:3 74:882 83:882 84:882 87:1.0 90:31.5 91:18.375 92:3.0 93:3.0 94:8.0 95:8.0 96:0.375 97:0.375 98:1.0 99:0.42857142857142855 100:0.21428571428571427 101:0.2857142857142857 102:0.16666666666666666 103:1.0 104:1149.0 105:743044.0 106:1149.0 107:743044.0 108:302068.5 109:379943446788.9167 110:302068.5 111:379943446788.9167
