Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestEpochWrite and TestEpochParse run failed #182

Open
tpolong opened this issue Nov 24, 2021 · 26 comments
Open

TestEpochWrite and TestEpochParse run failed #182

tpolong opened this issue Nov 24, 2021 · 26 comments
Assignees
@randallwhitman
Milestone
v2.3

Comments

@tpolong
Copy link

tpolong commented Nov 24, 2021

TestGeoJsonSerDe and TestEsriJsonSerDe both can't run.
image
image
image
@randallwhitman
Copy link
Contributor

Hi @tpolong - thanks for reporting this. I'm curious in what time zone was that run - in case it makes a difference.

@tpolong
Copy link
Author

tpolong commented Dec 2, 2021

I'm in the 8th East District

@randallwhitman
Copy link
Contributor

Sorry, I'm not sure what time-zone that is — please see - https://24timezones.com/time-zones

@tpolong
Copy link
Author

tpolong commented Dec 7, 2021

UTC+8

@randallwhitman randallwhitman self-assigned this Dec 14, 2021
@randallwhitman
Copy link
Contributor

Oh, "can't run" may mean a compilation problem rather than failed test - i may have misread it. (the image does not mean much to me.)

@tpolong
Copy link
Author

tpolong commented Dec 18, 2021 via email

@randallwhitman
Copy link
Contributor

randallwhitman commented Dec 18, 2021 •
edited
Loading

Even worse, the compile error is in BaseJsonSerde, in the product source itself.
spatial-framework-for-hadoop/hive/src/main/java/com/esri/hadoop/hive/serde/BaseJsonSerDe.java:[324,24] error: method getPrimitiveEpoch in class HiveShims cannot be applied to given types;
Indicates that it may have broken after testing when cleaning up for check-in, and that the binaries published for v2.2.0 may not match the source labeled with v2.2.0 tag.
[No, the compile error I saw Friday was due to an unfinished change in my working directory, not in v2.2.0 itself.]

@randallwhitman
Copy link
Contributor

Both TestEsriJsonSerDe and TestGeoJsonSerDe just passed for me.
I'm scheduled tests at varying times today, so I can see of the results vary by time of day (e.g. around 00:00 UTC), if in fact mvn test will run as scheduled by at(1).

@randallwhitman
Copy link
Contributor

Cross-reference #141 #165

@randallwhitman
Copy link
Contributor

randallwhitman commented Dec 21, 2021 •
edited
Loading

All tests passed for me, at all the following times, PST:
0030 0230 0330 0345 0400 0415 0430 0530 1045 1130 1230 1330 1430 1530 1545 1600 1615 1630 1830 2030 2230
I would need additional information in order to try to work any on this (details in text, not image only; and explanation in English/Spanish).

@tpolong
Copy link
Author

tpolong commented Dec 22, 2021

Test set: com.esri.hadoop.hive.serde.TestGeoJsonSerDe

Tests run: 12, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.108 sec <<< FAILURE!
TestEpochParse(com.esri.hadoop.hive.serde.TestGeoJsonSerDe) Time elapsed: 0 sec <<< FAILURE!
java.lang.AssertionError: expected:<1703> but was:<1702>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at com.esri.hadoop.hive.serde.TestGeoJsonSerDe.TestEpochParse(TestGeoJsonSerDe.java:168)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
TestEpochWrite(com.esri.hadoop.hive.serde.TestGeoJsonSerDe) Time elapsed: 0.001 sec <<< FAILURE!
java.lang.AssertionError: expected:<1703> but was:<1702>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at com.esri.hadoop.hive.serde.TestGeoJsonSerDe.TestEpochWrite(TestGeoJsonSerDe.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

@tpolong
Copy link
Author

tpolong commented Dec 22, 2021

Test set: com.esri.hadoop.hive.serde.TestEsriJsonSerDe

Tests run: 15, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.625 sec <<< FAILURE!
TestEpochParse(com.esri.hadoop.hive.serde.TestEsriJsonSerDe) Time elapsed: 0.005 sec <<< FAILURE!
java.lang.AssertionError: expected:<1703> but was:<1702>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at com.esri.hadoop.hive.serde.TestEsriJsonSerDe.TestEpochParse(TestEsriJsonSerDe.java:201)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
TestEpochWrite(com.esri.hadoop.hive.serde.TestEsriJsonSerDe) Time elapsed: 0 sec <<< FAILURE!
java.lang.AssertionError: expected:<1703> but was:<1702>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at com.esri.hadoop.hive.serde.TestEsriJsonSerDe.TestEpochWrite(TestEsriJsonSerDe.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

@tpolong
Copy link
Author

tpolong commented Dec 22, 2021

Failed tests: TestEpochParse(com.esri.hadoop.hive.serde.TestEsriJsonSerDe): expected:<1703> but was:<1702>
TestEpochWrite(com.esri.hadoop.hive.serde.TestEsriJsonSerDe): expected:<1703> but was:<1702>
TestEpochParse(com.esri.hadoop.hive.serde.TestGeoJsonSerDe): expected:<1703> but was:<1702>
TestEpochWrite(com.esri.hadoop.hive.serde.TestGeoJsonSerDe): expected:<1703> but was:<1702>
Tests run: 49, Failures: 4, Errors: 0, Skipped: 0

@randallwhitman
Copy link
Contributor

Thanks for the test details. This is curious, why we see different results.
Did you test with default versions of Hive etc (that is, no -P arguments to mvn)?
I'm guessing this has something to do with time zones and/or HIVE-12192: Hive should carry out timestamp computations in UTC.

@randallwhitman
Copy link
Contributor

Noticing that the test results were posted about 23:30 PST, I've scheduled tests to run here at 23:30 UTC-8, to see if at that time of day it will repro on a computer in PST. I'm curious if it will repro in both morning and afternoon/evening in UTC+8.

@tpolong
Copy link
Author

tpolong commented Dec 23, 2021

微信截图_20211223164203

@tpolong
Copy link
Author

tpolong commented Dec 23, 2021

微信截图_20211223164242

@randallwhitman
Copy link
Contributor

I think that is saying that it happens with Hive-3.1? But does not mention what time of day the test was run?
(Here in PST additional test runs passed at 2315 2330 2345 0015.)

@tpolong
Copy link
Author

tpolong commented Dec 29, 2021 •
edited
Loading

System.out.print(new Date().getTime());
1640766137444

@randallwhitman
Copy link
Contributor

I think the next thing we want to figure out, is, whether the issue is with the tests only, versus whether the issue is in the Spatial-Framework product code itself.

@randallwhitman
Copy link
Contributor

In a VM set to UTC+0930 I repro 4 tests failed.

@randallwhitman
Copy link
Contributor

randallwhitman commented Jan 7, 2022 •
edited
Loading

TestEpochParse - epochFromWritable(fieldData) gives 1974-08-30 00:00 UTC when run in East TZ but 1974-08-31 00:00 UTC when run in West TZ, -Phive-3.1 -Phadoop-3.2
Maybe we're getting something unexpected from the Hive API.

@randallwhitman randallwhitman mentioned this issue Jan 7, 2022
Merged
@randallwhitman
Copy link
Contributor

I'm thinking to changing dependency defaults to up-to-date versions, then afterward circle back to whether to try to work around HIVE-12192 for Hive-3.0 and below.

@randallwhitman randallwhitman mentioned this issue Feb 23, 2022
Merged
@randallwhitman
Copy link
Contributor

Is the issue blocking analysis?
If a matter of build, the JAR file can be built with -DskipTests=true.

@tpolong
Copy link
Author

tpolong commented May 6, 2022

I have tested the latest code and it works fine

@randallwhitman
Copy link
Contributor

Thanks! I went ahead and merged the PR.

@randallwhitman randallwhitman added this to the v2.3 milestone May 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@randallwhitman randallwhitman

Labels
None yet
Projects
None yet
Milestone
v2.3
Development

No branches or pull requests
2 participants
@randallwhitman @tpolong