WIP Benchmarks between nuget versions

[Shazwazza]

This is a WIP for creating benchmarkdotnet tests to see performance (currently for testing between nuget packages).

I have asked about specifying different nuget package sources here dotnet/BenchmarkDotNet#922 (comment) (I feel like I had done this in some way when implementing that feature but can't remember now) since that will be needed to test the 'current' version. It is not possible to reference the 'current' projects and then also use 'WithNuget' because that results in Nuget errors.

Currently just added a single benchmark as an example, this is just the script from the home page. There's manual iterations specified in there so that we don't get warnings from BenchmarkDotNet about MultimodalDistribution or MinIterationTime since one less iterations are too fast to measure correctly.

This example benchmark isn't showing much since the results are nearly identical between versions, for example:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.400-preview-015151
  [Host]          : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT

IterationCount=15  LaunchCount=2  WarmupCount=10  

Method	Job	NuGetReferences	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
HomePageScript	4.8.0-beta00007	Lucene.Net.Analysis.Common 4.8.0-beta00007	207.5 ms	3.94 ms	5.65 ms	63000.0000	6000.0000	1000.0000	249.37 MB
HomePageScript	4.8.0-beta00008	Lucene.Net.Analysis.Common 4.8.0-beta00008	196.8 ms	6.20 ms	9.28 ms	60000.0000	9000.0000	2000.0000	250.74 MB
HomePageScript	4.8.0-beta00009	Lucene.Net.Analysis.Common 4.8.0-beta00009	208.8 ms	7.15 ms	9.54 ms	62000.0000	7000.0000	2000.0000	250.77 MB

[NightOwl888]

Thanks for putting this together.

I suspect the reason you are not seeing an improvement between versions is because you are using a RAMDirectory, which bypasses the codecs and some other components that we have improved.

I added benchmarks using the IndexFiles and SearchFiles from Lucene.Net.Demo as templates. These are set up to use a fixed seed to generate the same data over and over again so we don't have to add several megabytes of test files to the repo just to do a benchmark.

Index speed doesn't seem to have improved, but search speed has doubled since 4.8.0-beta00005 (the one with the most downloads on NuGet.org). Here are the results for .NET Core 3.1 on on Windows.

Indexing

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.301
  [Host]          : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT  [AttachedDebugger]
  4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT

InvocationCount=1  IterationCount=15  LaunchCount=2  
UnrollFactor=1  WarmupCount=10  

Method	Job	NuGetReferences	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
IndexFiles	4.8.0-beta00005	Lucene.Net.Analysis.Common 4.8.0-beta00005	734.9 ms	37.14 ms	55.58 ms	44000.0000	8000.0000	7000.0000	221.09 MB
IndexFiles	4.8.0-beta00006	Lucene.Net.Analysis.Common 4.8.0-beta00006	703.7 ms	13.92 ms	20.83 ms	44000.0000	8000.0000	7000.0000	221.07 MB
IndexFiles	4.8.0-beta00007	Lucene.Net.Analysis.Common 4.8.0-beta00007	705.3 ms	15.95 ms	23.38 ms	43000.0000	8000.0000	7000.0000	221 MB
IndexFiles	4.8.0-beta00008	Lucene.Net.Analysis.Common 4.8.0-beta00008	747.0 ms	42.43 ms	59.47 ms	44000.0000	8000.0000	7000.0000	221.39 MB
IndexFiles	4.8.0-beta00009	Lucene.Net.Analysis.Common 4.8.0-beta00009	725.3 ms	25.59 ms	36.70 ms	44000.0000	8000.0000	7000.0000	221.31 MB
IndexFiles	4.8.0-beta00010	Lucene.Net.Analysis.Common 4.8.0-beta00010	724.1 ms	15.41 ms	22.10 ms	44000.0000	8000.0000	7000.0000	221.23 MB

Searching

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.301
  [Host]          : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT  [AttachedDebugger]
  4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
  4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT

IterationCount=100  LaunchCount=3  WarmupCount=15  

Method	Job	NuGetReferences	Mean	Error	StdDev	Median	Gen 0	Gen 1	Gen 2	Allocated
SearchFiles	4.8.0-beta00005	Lucene.Net.Analysis.Common 4.8.0-beta00005,Lucene.Net.QueryParser 4.8.0-beta00005	171.18 ms	2.004 ms	10.283 ms	169.51 ms	9000.0000	1000.0000	-	41.35 MB
SearchFiles	4.8.0-beta00006	Lucene.Net.Analysis.Common 4.8.0-beta00006,Lucene.Net.QueryParser 4.8.0-beta00006	169.86 ms	2.666 ms	13.754 ms	168.14 ms	9000.0000	1000.0000	-	41.35 MB
SearchFiles	4.8.0-beta00007	Lucene.Net.Analysis.Common 4.8.0-beta00007,Lucene.Net.QueryParser 4.8.0-beta00007	173.67 ms	3.417 ms	17.595 ms	169.20 ms	9000.0000	1000.0000	-	41.26 MB
SearchFiles	4.8.0-beta00008	Lucene.Net.Analysis.Common 4.8.0-beta00008,Lucene.Net.QueryParser 4.8.0-beta00008	95.97 ms	1.178 ms	6.068 ms	95.84 ms	8666.6667	500.0000	-	40.33 MB
SearchFiles	4.8.0-beta00009	Lucene.Net.Analysis.Common 4.8.0-beta00009,Lucene.Net.QueryParser 4.8.0-beta00009	93.31 ms	0.776 ms	3.890 ms	93.60 ms	8666.6667	500.0000	-	40.33 MB
SearchFiles	4.8.0-beta00010	Lucene.Net.Analysis.Common 4.8.0-beta00010,Lucene.Net.QueryParser 4.8.0-beta00010	85.79 ms	0.307 ms	1.552 ms	85.70 ms	8666.6667	500.0000	-	40.19 MB

It would be good to set up a stage on Azure DevOps to run these (off by default) so we can see the results on different operating systems and target frameworks.

[NightOwl888]

I have rebased against the master branch so this is up to date again.

I also rearranged the project so all of the NuGet versions are defined in BuildConfigurations.cs. I also got it working (sort of) with a local build from source so it can be compared against the NuGet packages. Unfortunately, I couldn't find any documentation of what the defaults are for the properties of a Release build, so this appears to be basically building a Debug build with optimizations turned on.

As you can see by the above image, the performance of the LocalBuild is much worse than using the NuGet packages. Requires more research to set all of the Release properties when the configuration is LocalBuild.

This isn't quite up to the state which we can merge it, though - we still need a top-level benchmarks folder as mentioned in #349 and to reorganize the project as well as rename this from Lucene.Net.Tests.BenchmarkDotNet to something that doesn't include the word Tests, but it is a few steps closer to where we need to be and it is quicker to update all of the benchmarks with a new set of NuGet packages now.