forked from OpenPOWER-BigData/HDP2.5-tez
-
Notifications
You must be signed in to change notification settings - Fork 0
/
HDP-CHANGES.txt
238 lines (229 loc) · 18.4 KB
/
HDP-CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
Changes only in Hortonworks github and not in Apache Tez release 0.7.0 and hence not covered by CHANGES.txt.
INCOMPATIBLE CHANGES
NEW FEATURES
RMP-4813. Better handling of AM failures in Tez Job Commit stage
IMPROVEMENTS
TEZ-3382. (BUG-63319) Tez analyzer: Should be resilient to new counters.
TEZ-3379. (BUG-63224) Tez analyzer: Move sysout to log4j.
TEZ-3376. (BUG-62940) Fix groupId generation to account for dagId starting with 1.
TEZ-3374. (BUG-62940) Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
TEZ-3357. (BUG-62940) Change TimelineCachePlugin to handle DAG grouping.
TEZ-3337. (BUG-61892) Do not log empty fields of TaskAttemptFinishedEvent to avoid confusion.
TEZ-3331. (BUG-58448) Add operation specific HDFS counters to ATS.
TEZ-1248. (BUG-29880) Reduce slow-start should special case 1 reducer runs.
TEZ-3327. (BUG-61685) ATS Parser: Populate config details available in dag.
TEZ-3308. (BUG-52060) Add counters to capture input split length.
TEZ-3223. (BUG-61612) Support a NullHistoryLogger to disable history logging if needed.
TEZ-3286. (BUG-59271) Allow clients to set processor reserved memory per vertex (instead of per container).
TEZ-3291. (BUG-60681) Optimize splits grouping when locality information is not available.
TEZ-3256. (BUG-59490) [Backport HADOOP-11032] Remove Guava Stopwatch dependency
TEZ-3107. (BUG-59501) tez-tools: Log warn msgs in case ATS has wrong values (e.g startTime > finishTime).
TEZ-3175. (BUG-59493) Add tez client submit host.
TEZ-3202. (BUG-59499) Reduce the memory need for jobs with high number of segments.
TEZ-3147. (BUG-57591) Intermediate mem-to-mem: Fix early exit when only one segment can fit into memory.
TEZ-3196. (BUG-55535) java.lang.InternalError from decompression codec is fatal to a task during shuffle.
TEZ-2756. (BUG-54600) MergeManager close should not try merging files on close if invoked after a shuffle exception.
TEZ-2863. (BUG-54598) Container, node, and logs not available in UI for tasks that fail to launch
TEZ-3173. (BUG-54700) Update Tez AM REST APIs for more information for each vertex.
TEZ-3192. (BUG-55442) IFile#checkState creating unnecessary objects though auto-boxing.
TEZ-1911. (BUG-54604) MergeManager's unconditionalReserve() should check for memory limits before allocating.
TEZ-3103. (BUG-54696) Shuffle can hang when memory to memory merging enabled
BUG-54430. Backport analyzers in tez-tools.
TEZ-3155. (BUG-52552) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits.
TEZ-2468. Change the minimum Java version to Java 7.
TEZ-3017. HistoryACLManager does not have a close method for cleanup
TEZ-1314. (BUG-48777) Port MAPREDUCE-5821 to Tez.
TEZ-2949. (BUG-47794) Allow duplicate dag names within session for Tez.
TEZ-2936. (BUG-46603) Support HDFS-based Timeline writer.
TEZ-2910. (BUG-46552) Tez should invoke HDFS Client API to set up caller context.
TEZ-2896. (BUG-46223) Fix thread names used during Input/Output initialization.
TEZ-2758. (BUG-45326) Remove append API in RecoveryService after TEZ-1909
TEZ-2561. (BUG-45379) Port for TaskAttemptListenerImpTezDag should be configurable
TEZ-2781. (BUG-43591) Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails
TEZ-2874. (BUG-44456) Improved logging for caller context.
TEZ-2868. (BUG-44456) Fix setting Caller Context in Tez Examples.
TEZ-2851. (BUG-44456) Support a way for upstream applications to pass in a caller context to Tez.
TEZ-2844. (BUG-44767) Backport TEZ-2775 to branch-0.7. Improve and consolidate logging in Runtime components.
TEZ-2719. (BUG-44767) Consider reducing logs in unordered fetcher with shared-fetch option
TEZ-2830. (BUG-44452) Backport TEZ-2774 to branch-0.7. Improvements to logging in the AM and part of the runtime.
TEZ-2812. Tez UI: Update task & attempt tables while in progress.
TEZ-2876. Tez UI: Update vertex, task & attempt details page while in progress.
TEZ-2817. Tez UI: update in progress counter data for the dag vertices and tasks table.
TEZ-2813. Tez UI: add counter data for rest api calls to AM Web Services v2.
TEZ-2789. (BUG-44090) Backport events added in TEZ-2612 to branch-0.7
TEZ-2767. (BUG-41183). Make TezMxBeanResourceCalculator the default resource calculator
TEZ-2461. tez-history-parser compile fails with hadoop-2.4.
TEZ-2076. Tez framework to extract/analyze data stored in ATS for specific dag.
BUG-43319. Port initial patch for ATS v1.5 from TEZ-2628
TEZ-2662. Provide a way to check whether AM or task opts are valid and error if not.
BUG-56245. Replace tez-ui v1 with v2 in apache master and erie branch. tez-ui copied from master#4389ce8cf368c913757a95788c9c792b2a71f37f
TEZ-3233. (BUG-53624) Tez UI: Have LLAP information reflect in Tez UI
TEZ-3086. (BUG-56627) Tez UI: Backward compatibility changes
TEZ-3254. (BUG-54441) Tez UI: Consider downloading Hive/Pig explain plans
TEZ-3255. (BUG-56044) Tez UI: Hide swimlane while displaying running DAGs from old versions of Tez
BUG-57985. Tez UI: Revert changes added by BUG-57728
TEZ-3259. (BUG-57728) Tez UI: Build issue - File saver package is not working well with bower
TEZ-3262. (BUG-57728) Tez UI : zip.js is not having a bower friendly versioning system
TEZ-3063. (BUG-30617) Tez UI : Display Input, Output, Processor, Source and Sink configurations under a vertex
TEZ-3281. (BUG-43252) Tez UI: Swimlane improvements
TEZ-3264. (BUG-57683) Tez UI: UI discrepancies
TEZ-3292. (BUG-57683) Tez UI: UTs breaking with timezone change
TEZ-3288. (BUG-58622) Tez UI: Display more details in the error bar
TEZ-3318. (BUG-58431) Tez UI: Polling is not restarted after RM recovery
TEZ-3329. (BUG-61330) Tez ATS data is incomplete for a vertex which fails or gets killed before initialization
TEZ-3333. (BUG-61915) Tez UI: Handle cases where Vertex/Task/Task Attempt data is missing
TEZ-3370. (BUG-62946) Tez UI: Display the log link as N/A if the app does not provide a log link
TEZ-3398. (BUG-63709) Tez UI: Bread crumb link to Application from Application details dag/configuration tab is broken
BUG FIXES
TEZ-2545. (BUG-62817) It is not necessary to start the vertex group commit when DAG is in TERMINATING
TEZ-2857. (BUG-62134) Fix flakey tests in TestDAGImpl.
TEZ-3314. (BUG-52060) Double counting input bytes in MultiMRInput.
TEZ-3325. (BUG-55372) Flaky test in TestDAGImpl.testCounterLimits.
TEZ-3293. (BUG-61394) Fetch failures can cause a shuffle hang waiting for memory merge that never starts.
TEZ-3305. (BUG-60245) TestAnalyzer fails with Hadoop 2.7.
TEZ-3304. (BUG-60245) TestHistoryParser fails with Hadoop 2.7.
TEZ-3294. (BUG-56680) DAG.createDag() does not clear local state on repeat calls.
TEZ-3297. (BUG-59986) Deadlock scenario in AM during ShuffleVertexManager auto reduce.
TEZ-3131. (BUG-59496) Support a way to override test_root_dir for FaultToleranceTestRunner.
TEZ-2579. (BUG-60016) Incorrect comparison of TaskAttemptId
TEZ-3295. (BUG-59614) TestOrderedWordCount should handle relative input/output paths
TEZ-3224. (BUG-59492) User payload is not initialized before creating vertex manager plugin.
TEZ-2846. (BUG-59606) Flaky test: TestCommit.testVertexCommit_OnDAGSuccess.
TEZ-3193. (BUG-59491) Deadlock in AM during task commit request.
TEZ-3036. (BUG-59497) Tez AM can hang on startup with no indication of error
TEZ-3258. (BUG-59489) Jvm Checker does not ignore DisableExplicitGC when checking JVM GC options.
TEZ-3052. (BUG-59498) Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED
TEZ-3289. (BUG-59345) Tez Example MRRSleep job does not set Staging dir correctly on secure cluster.
TEZ-3203. (BUG-59222) DAG hangs when one of the upstream vertices has zero tasks.
TEZ-3213. (BUG-56815) Uncaught exception during vertex recovery leads to invalid state transition loop.
TEZ-3276. (BUG-58654) Tez Example MRRSleep job fails when tez.staging-dir fs is not same as default FS.
TEZ-2097. (BUG-47918) TEZ-UI Add dag logs backend support
TEZ-2311. (BUG-57517) AM can hang if kill received while recovering from previous attempt
TEZ-3114. (BUG-54603) Shuffle OOM due to EventMetaData flood (jlowe)
TEZ-1961. (BUG-30204) Remove misleading exception "No running dag" from AM logs
TEZ-3105. (BUG-54597) TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures.
TEZ-2988. (BUG-54605) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier.
TEZ-3177. (BUG-47365) Non-DAG events should use the session domain or no domain if the data does not need protection.
TEZ-3189. (BUG-54666) Pre-warm dags should not be counted in submitted dags count by DAGAppMaster.
TEZ-3126. (BUG-54539) Log reason for not reducing parallelism.
TEZ-2886. (BUG-54426) Ability to merge AM credentials with DAG credentials.
TEZ-2963. (BUG-54425) RecoveryService#handleSummaryEvent exception with HDFS transparent encryption + kerberos authentication.
TEZ-3166. (BUG-31891) Counters aren't fully updated and sent for failed tasks
TEZ-3066. (BUG-54432) TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services.
BUG-54420. Fix Tez to handle changes for BUG-54323.
TEZ-3128. (BUG-53461) Avoid stopping containers on the AM shutdown thread.
TEZ-3156. (TEZ-87) Tez client keeps trying to talk to RM even if RM does not know about the application.
TEZ-3137. (BUG-52985) tez task failed with illegal state exception
TEZ-3123. (BUG-52871) Containers can get re-used even with conflicting local resources.
TEZ-3117. (BUG-52605) Deadlock in Edge and Vertex code
TEZ-2900. (BUG-51991) Ignore V_INPUT_DATA_INFORMATION when vertex is in Failed/Killed/Error
TEZ-2307. (BUG-51891) Possible wrong error message when submitting new dag
TEZ-2898. (BUG-50966) tez tools : swimlanes.py is broken.
TEZ-3037. (BUG-50911) History URL should be set regardless of which history logging service is enabled.
TEZ-3032. (BUG-51020) Incorrect start time in different events for DAG history events.
TEZ-3025. (BUG-51013) InputInitializer creation should use the dag ugi.
TEZ-2997. (BUG-49789) Tez UI: Support searches by CallerContext ID for DAGs.
TEZ-2995. (BUG-49802) Timeline primary filter should only be on callerId and not type.
TEZ-2975. (BUG-49421) Bump up apache commons dependency.
TEZ-2970. (BUG-49172) Re-localization in TezChild does not use correct UGI.
TEZ-2968. (BUG-49071) Counter limits exception causes AM to crash.
TEZ-2960. (BUG-48751) Tez UI: Move hardcoded url namespace to the configuration file
TEZ-2946. (BUG-45381) Tez UI: At times RM return a huge error message making the yellow error bar to fill the whole screen
TEZ-2947. (BUG-45381) Tez UI: Timeline, RM & AM requests gets into a consecutive loop in counters page without any delay
TEZ-2923. (BUG-45381) Tez Live UI counters view empty for vertices, tasks, attempts
TEZ-2933. (BUG-47675) Tez UI: Load application details from RM when available
TEZ-2908. (BUG-46445) Tez UI: Errors are logged, but not displayed in the UI when AM fetch fails
TEZ-2930. (BUG-47756) Tez UI: Parent controller is not polling at times
TEZ-2929. (BUG-47407) Tez UI: Dag details page displays vertices to be running even when dag have completed
TEZ-2927. (BUG-45378) Tez UI: Graciously fail when system-metrics-publisher is disabled
TEZ-2915. (BUG-45266) Tez UI: Getting back to the DAG details page is difficult
TEZ-2895. (BUG-45385) Tez UI: Add option to enable and disable in-progress
TEZ-2894. (BUG-45386) Tez UI: Disable sorting for few columns while in progress. Display an alert on trying to sort them
TEZ-2893. (BUG-45384) Tez UI: Retain vertex info displayed in DAG details page even after completion
TEZ-2878. (BUG-45378) Tez UI: AM error handling - Make the UI handle cases in which AM returns unexpected/no data
TEZ-2866. (BUG-45394) Tez UI: Newly added columns wont be displayed by default in tables
TEZ-2909. (BUG-46654) Tez UI: Application link in All DAGs table is disable when applicationhistory is unavailable
TEZ-2553. (BUG-46924) Tez UI: Tez UI Nits
TEZ-2899. (BUG-46926) Backport graphical view fix from TEZ-2899 to 2.3 maint
TEZ-2882. (BUG-45880) Consider improving fetch failure handling.
TEZ-2907. (BUG-46604) NPE in IFile.Reader.getLength during final merge operation.
TEZ-2850. (BUG-46495) Tez MergeManager OOM for small Map Outputs.
TEZ-2787. (BUG-46148) Tez AM should have java.io.tmpdir=./tmp to be consistent with tasks.
TEZ-2538. (BUG-45445) ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple partitions
TEZ-2885. (BUG-46163) Remove counter logs from AMWebController.
TEZ-2887. (BUG-45816) Tez build failure due to missing dependency in pom files.
TEZ-2829. (BUG-45682) Tez UI: minor fixes to in-progress update of UI from AM
TEZ-2436. (BUG-45768) Tez UI: Add cancel button in column selector
TEZ-2398. (BUG-45054) Flaky test: TestFaultTolerance
TEZ-2808. (BUG-45105) Race condition between preemption and container assignment
TEZ-2853. (BUG-45051) Tez UI: task attempt page is coming empty
TEZ-2834. (BUG-44763) Make Tez preemption resilient to incorrect free resource reported by YARN
TEZ-2816. (BUG-44763) Preemption sometimes does not respect heartbeats between preemptions
TEZ-814. (BUG-44763) Improve heuristic for determining a task has failed outputs
TEZ-2847. (BUG-44929) Tez UI: Task details doesn't gets updated on manual refresh after job complete
TEZ-2483. (BUG-44505) TEZ-2843. Tez UI: Show error if in progress fails due to AM not reachable
TEZ-2842. (BUG-44895) Tez UI: Update Tez App details page while in-progress
TEZ-2663. (BUG-44498) SessionNotRunning exceptions are wrapped in a ServiceException from a dying AM.
TEZ-2825. (BUG-44401) Report progress in terms of completed tasks to reduce load on AM for Tez UI
TEZ-2660. (BUG-44284) Tez UI: need to show application page even if system metrics publish is disabled.
TEZ-2780. (BUG-43253) Tez UI: Update All Tasks page while in progress
TEZ-2792. (BUG-43253) addendum fix build failure for java 6
TEZ-2792. (BUG-43253) Add AM web service API for tasks
TEZ-2807. (BUG-44090) Log data in the finish event instead of the start event
TEZ-2752. (BUG-44090) logUnsuccessful completion in Attempt should write original finish time to ATS
TEZ-2761. (BUG-44080) addendum fix build failure for java 6
TEZ-2766. (BUG-44080) Tez UI: Add vertex in-progress info in DAG details
TEZ-2761. (BUG-44080) Tez UI: update the progress on the dag and vertices pages with info from AM
TEZ-2754. (BUG-44080) Tez UI: StartTime & EndTime is not displayed with right format in Graphical View
TEZ-2211. (BUG-44080) Tez UI: Allow users to configure timezone
TEZ-2291. (BUG-44080) TEZ UI: Improper vertex name in tables.
TEZ-2549. (BUG-44080) Reduce Counter Load on the Timeline Server
TEZ-2768. (BUG-41800). Log a useful error message when the summary stream cannot be closed when shutting
down an AM.
TEZ-2745. (BUG-43220) ClassNotFound in InputInitializer causes AM to crash
TEZ-2602. (BUG-40970) Throwing EOFException when launching MR job
TEZ-2575. (BUG-40970) Handle KeyValue pairs size which do not fit in a single block in PipelinedSorter
TEZ-2198. (BUG-40970) Fix sorter spill counts
TEZ-2440. (BUG-40970) Sorter should check for indexCacheList.size() in flush()
TEZ-2742. VertexImpl.finished() terminationCause hides member var of the
same name
TEZ-2732. (BUG-43010) DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers
TEZ-2734. (BUG-43108) Add a test to verify the filename generated by OnDiskMerge.
TEZ-2687. (BUG-42805). Tez should release/kill all held containers before stopping services during the shutdown phase
TEZ-2635. (BUG-42217). Limit number of attempts being downloaded in unordered fetch.
TEZ-2629. (BUG-43012). LimitExceededException in Tez client when DAG has exceeds the default max counters.
TEZ-2730. tez-api missing dependency on org.codehaus.jettison for json.
TEZ-2552. (BUG-42823) CRC errors can cause job to run for very long time in large jobs.
TEZ-2636. (BUG-41461) MRInput and MultiMRInput should work for cases when there are 0 physical inputs.
BUG-39836 (TEZ-2568). auto_sortmerge_join_5 fails in Tez mode
BUG-36811. Disable swimlanes tab in Tez UI
TEZ-2548. TezClient submitDAG can hang if the AM is in the process of shutting down.
TEZ-2475. Fix a potential hang in Tez local mode caused by incorrectly handled interrupts.
TEZ-2567. Tez UI: download dag data does not work within ambari
TEZ-2554. Tez UI: View log link does not correctly propagate login crendential to read log from yarn web.
TEZ-2547. Tez UI: Download Data fails on secure, cross-origin clusters
TEZ-2546. Tez UI: Fetch hive query text from timeline if dagInfo is not set.
TEZ-2513. Tez UI: Allow filtering by DAG ID on All dags table.
TEZ-2541. DAGClientImpl enable TimelineClient check is wrong.
TEZ-2539. Tez UI: Pages are not updating in IE.
TEZ-2535. Tez UI: Failed task attempts link in vertex details page is broken.
TEZ-2489. Disable warn log for Timeline ACL error when tez.allow.disabled.timeline-domains set to true.
TEZ-2528. Tez UI: Column selector buttons gets clipped, and table scroll bar not visible in mac.
TEZ-2391. TestVertexImpl timing out at times on jenkins builds.
TEZ-2509. YarnTaskSchedulerService should not try to allocate containers if AM is shutting down.
TEZ-2527. Tez UI: Application hangs on entering erroneous RegEx in counter table search box
TEZ-2523. Tez UI: derive applicationId from dag/vertex id instead of relying on json data
TEZ-2505. PipelinedSorter uses Comparator objects concurrently from multiple threads.
TEZ-2504. Tez UI: tables - show status column without scrolling, numeric 0 shown as Not available
TEZ-2478. Move OneToOne routing to store events in Tasks.
TEZ-2482. Tez UI: Mouse events not working on IE11
TEZ-1529. ATS and TezClient integration in secure kerberos enabled cluster.
TEZ-2481. Tez UI: graphical view does not render properly on IE11
TEZ-2474. The old taskNum is logged incorrectly when parallelism is changed
TEZ-2460. Temporary solution for issue due to YARN-2560
TEZ-2455. Tez UI: Dag view caching, error handling and minor layout changes
TEZ-2453. Tez UI: show the dagInfo is the application has set the same.
TEZ-2447. Tez UI: Generic changes based on feedbacks.
TEZ-2409. Allow different edges to have different routing plugins
TEZ-3101. (BUG-53754) Tez UI: Task attempt log link doesn't have the correct protocol.
TEZ-2560. (BUG-55672) fix tex-ui build for maven 3.3+