filter2 command is too slow

[y9c]

Compared with filter command or awk, fiter2 command is much slower, especially for rule with multiple conditions.

It might be relative to this function in the for-loop, which repeatedly parsing the expression.

csvtk/csvtk/cmd/filter2.go

Lines 370 to 376 in 9407f73

	// evaluate
	if containCustomFuncs {
	expression, err = govaluate.NewEvaluableExpressionWithFunctions(filterStr1, functions)
	} else {
	expression, err = govaluate.NewEvaluableExpression(filterStr1)
	}
	checkError(err)

[shenwei356]

Yes, I noticed that. It is slow :(

[y9c]

Can we move the Expression parsing function outside the for-loop and run it only once?

[shenwei356]

It is slow, but it must be done like that. Cause filterStr1 is different in each iteration.

[y9c]

Why filterStr1 is different? Can we cache the parsed results?

[shenwei356]

It's the expression, like '$age > 18', the $age needs to be replaced with the value of each row.

[y9c]

Yes. I mean can we parsed the expression as something like '$1>18' and reuse the code of the filter command to deal with the computation afterward

[shenwei356]

parsed the expression as something like '$1>18' and reuse the code of the filter command
I don't think so.

God, it's really slow~ I used it a lot recently. Have to improve it, when I have time ~

[shenwei356]

It's fixed. Please update it to v0.31.1. https://github.com/shenwei356/csvtk/releases/tag/v0.31.1

The problem is in the parameter preparation step, not the expression evaluation. d20aa89