Error using Lucene.Net.Facet 4.8.0-beta00005 with Xamarin.iOS #273
Comments
|
Thanks for the report. I took a look and it really isn't very clear what the future holds for this problem. According to Microsoft this is a current issue and it isn't clear whether they intend to address it. However, according to this page it sounds like this should be currently supported. Microsoft's documentation was apparently copied from this page, which was last updated in 2009. Then again, according to this comment:
Since I can't seem to find a way to detect whether the AOT compilation is enabled and there are clear advantages (optimizations) for calling System.Collections.Generic.EqualityComparer.Default on other platforms, I ended up wrapping it in a new comparer Lucene.Net.Support.EqualityComparer.Default, which attempts to call System.Collections.Generic.EqualityComparer.Default the first time and if there is an exception it will store a nullable boolean in a static field to indicate whether value types are supported, or null to indicate the check hasn't yet been done. After the initial attempt, it will simply call the correct comparer depending on the state of the boolean. That should fix the issue and also cover the case where it is eventually patched. This could be improved if there were a specific exception to catch (you didn't include it in the stack trace) to narrow down the field - right now it catches when any exception is thrown and the type is a value type, but allows all other exceptions to be thrown. Version 4.8.0-ci0000001189 on the CI Feed has the update - let us know if that fixes the problem. We'd appreciate a pull request to update the Powershell build script so it works on Xamarin.iOS, Xamarin.Android, and UWP, and then we can set it up in TeamCity to continually test these platforms and help us track down issues (if you have the time, we should also do Mono and Xamarin.Mac). Unfortunately, I don't currently have access to all of these platforms nor the space that would be required to virtualize/emulate them in order to ensure the script works, and doing it all on the TeamCity server would be slow going. The test task begins here - we would need an update to that and might need to update the tasks to install platforms, if required (we had to do that for dotnet core because the TeamCity servers don't necessarily have the version we need installed and end users might not, as well). To test, the build/build.ps1 script is run with the following command from TeamCity: Import-Module .\build\psake.psm1; Invoke-Psake .\build\build.ps1 -Task Test -properties @{frameworks_to_test='netcoreapp2.0';backup_files='false';prepareForBuild='false'}
To build, pack and then test (which we use from the command line), the command looks like: Import-Module .\build\psake.psm1; Invoke-Psake .\build\build.ps1 Default,Test -properties @{configuration='Release'} -parameters @{ packageVersion='4.8.0-beta00006';version='0.0.0' }
We are using PSake as a task manager, and the documentation for it is here. Also, we currently only have a build.bat file to launch the command manually, which needs to be translated into a build.sh to run cross-platform. Note that TeamCity does not use this file, it is only for end users who want to build/test. Also note this file is automatically re-generated during a release by the PSake script so the user only needs to type "build" on the command line and the version of the current release will be baked in - we would want to do that for a build.sh file, too. by nightowl888 |
|
Tank you for your support. I downloaded and tested the update 4.8.0-ci0000001189. Unfortunately I have the same error: Attempting to JIT compile method 'Lucene.Net.Support.LurchTable`2:InternalInsert> (int,Lucene.Net.Facet.Taxonomy.FacetLabel,int&,Lucene.Net.Support.LurchTable`2/Add2Info&)' while running in aot-only mode. See https://developer.xamarin.com/guides/ios/advanced_topics/limitations/ for more information. at Lucene.Net.Support.LurchTable`2[TKey,TValue].Insert[T] (TKey key, T& value) <0x25c3094 + 0x000e0> in :0
The second part of message is different from the second part of the first message because we changed our porgram but I think the problem is the same. At the moment I can't obtain a better stack trace: any suggestion? For the update of Powershell build script i'm studying ... Thank you in advance, Enrico Caltran +393357485560 [email protected] by enycaltran |
My suggestion would be to step through some test code to find out which of these 3 lines are throwing exceptions and what they are, something like: try { var comparer = System.Collections.Generic.EqualityComparer<int>.Default; I was assuming it was the first line, but it is apparently the second and/or 3rd line that are throwing, and some feedback as to which of the lines throw exceptions and what type of exception is being thrown would be helpful.
So, having some tests to run on AOT to provide feedback on whether the fix is complete would be extremely helpful. by nightowl888 |
|
Hello, In case if you want I'll send you the test code used. Thank you in advance. by enycaltran |
|
Enrico, Thanks for the feedback. A Google search indicates several people who have run into this issue and most of them seem to indicate that changing the compilation settings will fix the issue. In particular, see:
While many of the answers seem to indicate that a System.ExecutionEngineException is thrown from .NET, when trying to catch the exception I get a compile warning:
Therefore, I would suggest trying the solutions in the above answers on the 4.8.0-beta00005 build to see if you can fix this using the compilation settings, as some of the documentation and answers seem to indicate this issue has already been fixed on AOT. The fact the exception thrown is marked obsolete also seems to indicate this is not the right way to fix this issue. IMPORTANT: 4.8.0-beta00004 was compiled with optimizations disabled, so there is a high likelihood that it will not work with AOT. Make sure you are using 4.8.0-beta00005. If you are still having the issue, please put together a small standalone project with code that causes the exception to happen along with step by step instructions about how to make it occur so we can try to reproduce the issue here.
Thanks. But in this case we are not calling native functions, so this doesn't seem like a solution to this particular issue. by nightowl888 |
|
Hello, We tried to apply every suggestion indicated in links, but we have the same issue.
We prepared a little Xamarin Solution where we replicated the issue. We attach the zip with the VisualStudio solution. Click on "Lucene Button" and then start Core.Lucene.CoreLucene.TestFaceting() in CoreLucene.cs In Uwp, Andorid and Ios Simulator we dont have any problem. In Ios device IOS we have the usual exception of topic when we do:
facetResult = facets.GetTopChildren(10, "Author");
NB: if build the AppTestLucene.iOS with property Link Behavior: Link All (insead Lin Framework SDKs Only), I obtain 2 new issues:
IndexWriterConfig config = new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer); I receive a new very strange Exception: Unhandled Exception: System.ArgumentException: Codec 'Lucene46' cannot be loaded. If the codec is not in a Lucene.Net assembly, you must subclass DefaultCodecFactory and call PutCodecType() or ScanForCodecs() from the Initialize() method. So at the moment, if you want to run ord debug the application you must compile with Link Behavior = Framework SDKs Only
Thank you in advance,
Enrico. by enycaltran |
|
Hello, Do you have some news? Thank you in advance,
Enrico. by enycaltran |
|
Hello, excuse me if i'm insinsting, but i must understand if I can continue along this road or I must change idea. Can you kindly answer? I'm waiting from about 20 days.... Thank you in advance,
Enrico. by enycaltran |
|
Enrico, Sorry for the long delay. The issue here seems to be that we currently use Reflection to load the Codec, DocValuesFormat, and PostingsFormat types and apparently not all platforms have complete implementations of Reflection and silently fail. However, as the error message says, there is an API that can be used to load the types manually by subclassing factories for each of those 3 types.
Here is an example of subclassing these types to provide codecs. It is meant to be able to add your own codecs, but you can also use it to workaround the issue of the codecs not loading until the next beta where this will be fixed. private class MyCodecFactory : DefaultCodecFactory { // These are all of the codec types that are in // Lucene.Net.dll. You only need the first one unless // you need legacy support for Lucene.Net 3.x or // want to build your own codec. private static Type[] localCodecTypes = new Type[] { typeof(Lucene46.Lucene46Codec), #pragma warning disable 612, 618 typeof(Lucene3x.Lucene3xCodec), // Optimize 3.x codec over < 4.6 codecs typeof(Lucene45.Lucene45Codec), typeof(Lucene42.Lucene42Codec), typeof(Lucene41.Lucene41Codec), typeof(Lucene40.Lucene40Codec), #pragma warning restore 612, 618 }; The pragma statements aren't strictly necessary, they are just to suppress the obsolete warnings for those types. There are many more examples on setting up these factories in the tests CodecFactory, DocValuesFormatFactory, PostingsFormatFactory. by nightowl888 |
|
Hi Shad I seem to have a slightly different error for what seems to be the same underlying problem. Running the sample from Enrico works fine on Android and the IOS emulators, but break runtime on a physical IOS device.
================================================================= Native Crash Reporting ================================================================= Got a SIGSEGV while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. =================================================================
I found the cause of the problem to be the implementation of the LRUHashMap using LurchTable. The LurchTable class is violating the restrictions in Mono for IOS. I simply replaced the LurchTable with a randomly selected LruCache - LruCacheNet https://www.nuget.org/packages/LruCacheNet/ and this solved the problem.
Any chance you could fix this permanently? by borge |
|
Interesting. We chose LurchTable because it was seemingly the only thread safe LRU cache for .NET in existence at the time. We copied and pasted the code because at the time NOBODY supported .NET Standard yet. I would be happy to use it, but there is a compatibility issue as it doesn't support .NET Framework 4.5 or .NET Standard 1.6. I'd rather not copy and paste it into our project again, but have the owners update their project to support more targets than just .NET Standard 2.0. Worst case, I guess we could target it for .NET Standard 2.0 and use LurchTable for .NET Framework/.NET Standard 1.x, but it would be nice not to have to add conditional compilation for this. Have you verified that all of the Facet tests pass with this LRU cache? by nightowl888 |
|
I must admit, I only did a quick and dirty hack to verify that the problem was in the LurchTable implementation. I have not carefully chosen an alternative, and I'm surly not technically competent to properly compare the different implementations out there, against the features needed in this project. With your knowledge of the necessary features, I has hoping you could find a propper alternative, that hopefully could support the correct frameworks. There seems to be a few candidates according to Google: lrucache c# And, no. I haven't verified that all of the Facet tests pass... by borge |
|
If you want to see what I tested, this is the changes I did in Lucene.Net.Facet/Taxonomy/LRUHashMap.cs: @@ -2,6 +2,7 @@ using System; using System.Collections; using System.Collections.Generic; using LruCacheNet; With colours: |
|
Come to think of it, the primary reason why LurchTable was chosen was not for thread safety, but for performance (it's all coming back to me now)....https://apache.markmail.org/search/?q=lucenenet+lurchtable#query:lucenenet%20lurchtable+page:1+mid:wtuyqd3sfre6532s+state:results. We had another implementation previously that looks a lot like the one you linked to (with lots of lock statements) and we had some complaints about how it performed. I would rather try to fix it so it works with Xamarin.iOS than to replace it, if possible.
Just out of curiosity, what happens if you provide a custom EqualityComparer instead of falling back to EqualityComparer.Default? Do note that LurchTable seems to have many hard coded references to EqualityComparer.Default, it doesn't consistently use the passed in comparer argument everywhere. I tried that before, but without any way to see how/why it is failing, it is like shooting in the dark. It would be helpful if you could provide a fix that works. One other possibility is to wrap your value type keys inside of reference types and use those in the declaration of LurchTable. For example: // This is pseudo-code - you will probably need to fix it up to compile private class ValueTypeRef //where TKey: struct { public TKey Value { get; set } by nightowl888 |
|
Here is the attempt I tried to do to fix the EqualityComparers for value types, as I am sure this goes beyond fixing LurchTable if that is actually the cause: 199fab0 I have since reverted all but the LRUHashMap constructor overload, as it doesn't belong in our codebase if it doesn't have a purpose. But, if you can change the comparer implementation to one that actually works on Xamarin.iOS and send it back to me or provide a PR, that would be helpful |
|
Hi Shad, we encountered the same issue about LurchTable reported by Axel, while using the facets on Xamarin iOS. If it can help, we've found this issue, where the same problem is discussed: mono/mono#7016 The problem is due to the the use, in LurchTable, of some structs that implement some generic interfaces. At the end of the discussion, they says here mono/sysdrawing-coregraphics#24 We tried the other fixes you proposed in the previous comments, but they didn't work. Do you suggest any other solution? |
|
Stefano, Thanks for the report. We used to use LruCache (or some similar class) and it was compatible with the facet functionality, but we got some complaints about performance. LurchTable was found to perform much better (on Windows, anyway), primarily due to the locking contention that is present in LruCache. Therefore, using LruCache as-is for any platforms that fully support LurchTable is not an acceptable trade-off due to the performance penalty. Note that although no users have reported problems with it, this issue affects some parts of Lucene.Net.QueryParser in addition to Lucene.Net.Facet. I agree fixing the LRU cache is something that needs to be done but so far there hasn't been enough information provided for us to fix it without taking a performance hit. Of course, it would definitely help if our tests were setup to run on Xamarin.iOS so we have a way to confirm that any change has a positive effect, which is something that is currently up-for-grabs (see GH-633). Adding testing support should be simpler to do now that we are setup on Azure DevOps via YAML configuration. If we cannot somehow patch LurchTable to make it work with Xamarin.iOS, the next best option would be to detect when we are running on Xamarin.iOS and swap the implementation to LruCache during initialization for only that case. I found some information on this StackOverflow answer (which coincidentally is one that I also answered) that seems like it might be a way to detect that specific case, especially if you click through to the related documents. So, it seems there is a way to detect Xamarin and a way to detect Device.iOS, and hopefully combining the two methods should give us the specific behavior we need to provide this swap only in the case where it is needed. It would be extremely helpful if you could verify that the combination of the above methods will indeed detect only Xamarin.iOS and not trigger on Xamarin in general or iOS in general. It would be even more helpful if you could setup our tests to run on Xamarin.iOS and provide tests we can run that confirm the platform detection will work. Do note that the general place where we do platform detection is in Lucene.Net.Util.Constants.cs. The current thinking is to clean up all of our collections (including those that we have absorbed from 3rd parties) and make them into 1st class features that can be used by end users (see GH-616). However, it goes beyond that - we have also absorbed a handful of Java-like types that really need to be polished and put into a separate, stable general library that can be utilized by Lucene.NET's dependencies to share interfaces, collections, and other compatibility functionality that hasn't yet debuted in the .NET ecosystem. We are sometimes running into issues where we cannot pass a type in library A to library B because there is no common interface in the BCL to pass it like there is in Java. While it would certainly be better to submit PRs and get the collection libraries up to speed as far as behavior, interfaces, and platform compatibility is concerned, I am becoming convinced that it is not likely going to happen. I reported back in 2016 to the maintainers of C5 that their sets don't support ISet and although they thought it might be a good idea to do, it still hasn't been done. I recently started working on a PR, but found that C5 would require breaking API changes to their dictionaries (or alternatively a compatibility wrapper) just to make them compatible with common collection interfaces in the BCL. Furthermore, they are already dropping support for .NET Framework 4.5 and .NET Standard 1.x. So, we are likely going to have to build our own library of collections in order to get the level of compatibility, stability, and most importantly end-user usability we will need. Anyway, any assistance that is provided to help us get to the bottom of this without it costing us dearly in terms of performance would be appreciated. by nightowl888 |
|
Hi Shad, a way to detect at runtime if the library is running on Xamarin.iOS is: var assembly = Assembly.GetEntryAssembly()?.GetReferencedAssemblies().FirstOrDefault(x => x.Name == "Xamarin.iOS"); return assembly != null;
by francescob |
|
Thanks for that. Although, I was able to get a bit more information about what specifically not working with LurchTable. It turns out that simply changing the AddInfo struct to a class will solve the problem. However, since doing so will likely negatively affect performance to some extent, I would like to try to find another solution. Could someone please attempt to resolve this by de-nesting the AddInfo struct, and changing its signature to the following? struct AddInfo : ICreateOrUpdateValue
Do note that I am working on getting beta00007 out within the next few days. We'd like to include a fix for this, if possible. by nightowl888 |
|
Hi Shad, I de-nested the struct AddInfo and Add2Info changing the signature: struct AddInfo<TKey, TValue> : ICreateOrUpdateValue<TKey, TValue>;
struct Add2Info<TKey, TValue> : ICreateOrUpdateValue<TKey, TValue>;and it works on real device (ipad running iOS 13) by berardi |
|
Thank you. I have pushed the patch 4eb9e8d68cc18b10503e90241af995e03187dd06 to the master branch. If you have a chance, please download the NuGet packages from here or build the source to try it again with Lucene.Net.Facet. by nightowl888 |
|
This should now be resolved in Lucene.NET 4.8.0-beta00007. Could someone please confirm the problem no longer exists? by nightowl888 |
|
I suspect this issue may crop up again due to the new collections in J2N which Lucene.NET now uses. Although, there may be no issue for interfaces with 1 generic parameter, such as IEnumerable so we might be okay. Basically, we have copied the structure of the collections in System.Collections.Generic, which use nested structs as enumerators, and each implements a generic interface: public class LinkedDictionary { internal struct Enumerator : IEnumerator> {} } Could someone please do a test with J2N 2.0.0-beta-0001 on Xamarin.iOS to ensure this problem isn't going to creep in again? A good test would be to enumerate the LinkedDictanary (the dictionary, the keys, and the values). We have replaced most of the collections in Lucene.NET with the ones from J2N, so if this is a problem, it will affect more than just Lucene.Net.Facet in the next Lucene.NET release, it will affect practically every module. by nightowl888 |
|
Hi Shad, we are planning to take the tests in the next few days. We will let you know. by francescob |
|
Hi Shad, I create a new Xamarin.iOS project, imported J2N 2.0.0-beta-0001 and tested this code: var linkedDictionary = new LinkedDictionary<int, string>(); for (int i = 0 ; i < 100; i++) { linkedDictionary.Add(i, i.ToString()); } foreach(var entry in linkedDictionary) { Console.WriteLine($"Key: {entry.Key.ToString()} - Value: {entry.Value}"); } foreach(var entry in linkedDictionary.Keys) { Console.WriteLine($"Key: {entry.ToString()}"); } foreach(var entry in linkedDictionary.Values) { Console.WriteLine($"Value: {entry}"); } using(var enumerator = linkedDictionary.GetEnumerator()) { while(enumerator.MoveNext()) { var current = enumerator.Current; Console.WriteLine($"Key: {current.Key.ToString()} - Value: {current.Value}"); } } It works without problems. Let me know if it is enough or you need more tests. by francescob |
|
Is related to: GH-633 |
|
This has basically been held open because of NightOwl888/J2N#3, but given the fact this one reporter is no longer interested in a fix and MAUI is right around the corner (which hopefully addresses this), there doesn't seem to be much sense in spending time on this. |
I'm using Lucene.Net.Facet 4.8.0-beta00005 in a big Xamarin project.
With Xamarin.Android and Xamarin.UWP it's all right.
But With Xamarin.iOS on device (Ipad), i'm receiving this error:
Attempting to JIT compile method 'Lucene.Net.Support.LurchTable2:InternalInsert> (int,Lucene.Net.Facet.Taxonomy.FacetLabel,int&,Lucene.Net.Support.LurchTable`2/Add2Info&)' while running in aot-only mode. See https://developer.xamarin.com/guides/ios/advanced_topics/limitations/ for more information.
at Lucene.Net.Support.LurchTable2[TKey,TValue].Insert[T] (TKey key, T& value) <0x2570f48 + 0x000e0> in <063e095c95d945a4ace32ab83d1227eb#2ae0fea9ea4eacaef83bf2e9713bb8ea>:0 at (wrapper unknown) System.Object.gsharedvt_in() at Lucene.Net.Support.LurchTable2[TKey,TValue].AddOrUpdate (TKey key, TValue addValue, Lucene.Net.Support.KeyValueUpdate2[TKey,TValue] fnUpdate) <0x232824c + 0x0013b> in <063e095c95d945a4ace32ab83d1227eb#2ae0fea9ea4eacaef83bf2e9713bb8ea>:0 at Lucene.Net.Facet.Taxonomy.LRUHashMap2[TKey,TValue].Put (TKey key, TValue value) <0x2c487f8 + 0x0015b> in <79d3a7b905954d0993025c09c5d087ce#2ae0fea9ea4eacaef83bf2e9713bb8ea>:0 at Lucene.Net.Facet.Taxonomy.Directory.DirectoryTaxonomyReader.GetOrdinal (Lucene.Net.Facet.Taxonomy.FacetLabel cp) <0x2c51970 + 0x0019b> in <79d3a7b905954d0993025c09c5d087ce#2ae0fea9ea4eacaef83bf2e9713bb8ea>:0 at Lucene.Net.Facet.Taxonomy.Int32TaxonomyFacets.GetTopChildren (System.Int32 topN, System.String dim, System.String[] path) <0x2c481dc + 0x0008f> in <79d3a7b905954d0993025c09c5d087ce#2ae0fea9ea4eacaef83bf2e9713bb8ea>:0 at Login.MyMB.Lucene.Client.LuceneArticoliSearcher.GetListaArticoloXRicercaAvanzataConRicercaSemplice (System.Collections.Generic.List1[T] listParametri) <0x224add0 + 0x001bb> in <8f49891e0f0546e185aba7424d294ef7#2ae0fea9ea4eacaef83bf2e9713bb8ea>:0 at Login.MyMB.Lucene.Client.LuceneArticoliSearcher.GetListaArticoloConRicercaSemplice (System.Collections.Generic.List1[T] listParametri) <0x224afbc + 0x0009f> in <8f49891e0f0546e185aba7424d294ef7#2ae0fea9ea4eacaef83bf2e9713bb8ea>:0 at MyMB.Forms.RicercaLucene.RicercaArticoloLucene.GetListaArticoliXRicercaSemplice (Login.MyMB.Interface.IAmbiente ambiente, Login.MyMB.Lucene.Client.LuceneArticoliSearcher las, System.Collections.Generic.List`1[T] ListParametri, System.Boolean isAbilitataRicercaBarcode) <0xe47fc0 + 0x000e7> in :0 ...............................
At the link https://docs.microsoft.com/it-it/xamarin/ios/internals/limitations , I found the problem cause (I suppose...):
Value types as Dictionary Keys Using a value type as a Dictionary key is problematic, as the default Dictionary constructor attempts to use EqualityComparer.Default. EqualityComparer.Default, in turn, attempts to use Reflection to instantiate a new type which implements the IEqualityComparer interface. This works for reference types (as the reflection+create a new type step is skipped), but for value types it crashes and burns rather quickly once you attempt to use it on the device. Workaround: Manually implement the IEqualityComparer interface in a new type and provide an instance of that type to the Dictionary (IEqualityComparer) constructor.
So, what can I do? Thank you in advance, Enrico Caltran +393357485560 [email protected]
JIRA link - [LUCENENET-602] created by enycaltranThe text was updated successfully, but these errors were encountered: