+373 (69) 210 189 | info@fusionworks.md
bclose

Details for: Unable to run distributed shell on Yarn

Failed app in the list

Failed app

Logs unavailable

Logs from node manager
[bash] 3-09-18 08:41:10,557 INFO ipc.Server (Server.java:saslProcess(1342)) – Auth successful for appattempt_1379338026167_0125_000001 (auth:SIMPLE)
2013-09-18 08:41:10,563 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(456)) – Start request for container_1379338026167_0125_01_000001 by user hdfs
2013-09-18 08:41:10,563 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(481)) – Creating a new application reference for app application_1379338026167_0125
2013-09-18 08:41:10,564 INFO application.Application (ApplicationImpl.java:handle(430)) – Application application_1379338026167_0125 transitioned from NEW to INITING
2013-09-18 08:41:10,565 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) – USER=hdfs IP=XXXXXXXXXX OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1379338026167_0125 CONTAINERID=container_1379338026167_0125_01_000001
2013-09-18 08:41:10,630 INFO application.Application (ApplicationImpl.java:transition(277)) – Adding container_1379338026167_0125_01_000001 to application application_1379338026167_0125
2013-09-18 08:41:10,630 INFO application.Application (ApplicationImpl.java:handle(430)) – Application application_1379338026167_0125 transitioned from INITING to RUNNING
2013-09-18 08:41:10,630 INFO container.Container (ContainerImpl.java:handle(857)) – Container container_1379338026167_0125_01_000001 transitioned from NEW to LOCALIZING
2013-09-18 08:41:10,631 INFO localizer.LocalizedResource (LocalizedResource.java:handle(196)) – Resource hdfs://XXXXXXX:8020/user/hdfs/DistributedShell/125/AppMaster.jar transitioned from INIT to DOWNLOADING
2013-09-18 08:41:10,631 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(589)) – Created localizer for container_1379338026167_0125_01_000001
2013-09-18 08:41:10,637 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1014)) – Writing credentials to the nmPrivate file /hadoop/yarn/nmPrivate/container_1379338026167_0125_01_000001.tokens. Credentials list:
2013-09-18 08:41:10,645 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(468)) – Initializing user hdfs
2013-09-18 08:41:10,672 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(103)) – Copying from /hadoop/yarn/nmPrivate/container_1379338026167_0125_01_000001.tokens to /hadoop/yarn/usercache/hdfs/appcache/application_1379338026167_0125/container_1379338026167_0125_01_000001.tokens
2013-09-18 08:41:10,673 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(105)) – CWD set to /hadoop/yarn/usercache/hdfs/appcache/application_1379338026167_0125 = file:/hadoop/yarn/usercache/hdfs/appcache/application_1379338026167_0125
2013-09-18 08:41:10,964 INFO localizer.LocalizedResource (LocalizedResource.java:handle(196)) – Resource hdfs://XXXXXXX:8020/user/hdfs/DistributedShell/125/AppMaster.jar transitioned from DOWNLOADING to LOCALIZED
2013-09-18 08:41:10,964 INFO container.Container (ContainerImpl.java:handle(857)) – Container container_1379338026167_0125_01_000001 transitioned from LOCALIZING to LOCALIZED
2013-09-18 08:41:11,179 INFO container.Container (ContainerImpl This Site.java:handle(857)) – Container container_1379338026167_0125_01_000001 transitioned from LOCALIZED to RUNNING
2013-09-18 08:41:11,284 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(189)) – launchContainer: [nice, -n, 0, bash, -c, /hadoop/yarn/usercache/hdfs/appcache/application_1379338026167_0125/container_1379338026167_0125_01_000001/default_container_executor.sh] 2013-09-18 08:41:11,547 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:getNodeStatusAndUpdateContainersInContext(288)) – Sending out status for container: container_id { app_attempt_id { application_id { id: 125 cluster_timestamp: 1379338026167 } attemptId: 1 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2013-09-18 08:41:11,932 WARN nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(207)) – Exit code from container container_1379338026167_0125_01_000001 is : 1
2013-09-18 08:41:11,932 WARN nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(213)) – Exception from container-launch with container ID: container_1379338026167_0125_01_000001 and exit code: 1
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
at org.apache.hadoop.util.Shell.run(Shell.java:373)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-09-18 08:41:11,933 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(173)) –
2013-09-18 08:41:11,933 WARN launcher.ContainerLaunch (ContainerLaunch.java:call(289)) – Container exited with a non-zero exit code 1
2013-09-18 08:41:11,933 INFO container.Container (ContainerImpl.java:handle(857)) – Container container_1379338026167_0125_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE
2013-09-18 08:41:11,933 INFO launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(316)) – Cleaning up container container_1379338026167_0125_01_000001
2013-09-18 08:41:12,039 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(368)) – Deleting absolute path : /hadoop/yarn/usercache/hdfs/appcache/application_1379338026167_0125/container_1379338026167_0125_01_000001
2013-09-18 08:41:12,042 WARN nodemanager.NMAuditLogger (NMAuditLogger.java:logFailure(150)) – USER=hdfs OPERATION=Container Finished – Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1379338026167_0125 CONTAINERID=container_1379338026167_0125_01_000001
2013-09-18 08:41:12,042 INFO container.Container (ContainerImpl.java:handle(857)) – Container container_1379338026167_0125_01_000001 transitioned from EXITED_WITH_FAILURE to DONE
2013-09-18 08:41:12,042 INFO application.Application (ApplicationImpl.java:transition(320)) – Removing container_1379338026167_0125_01_000001 from application application_1379338026167_0125
2013-09-18 08:41:12,042 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(246)) – Considering container container_1379338026167_0125_01_000001 for log-aggregation
2013-09-18 08:41:12,094 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(337)) – Starting resource-monitoring for container_1379338026167_0125_01_000001
2013-09-18 08:41:12,094 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(347)) – Stopping resource-monitoring for container_1379338026167_0125_01_000001
2013-09-18 08:41:12,549 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:getNodeStatusAndUpdateContainersInContext(288)) – Sending out status for container: container_id { app_attempt_id { application_id { id: 125 cluster_timestamp: 1379338026167 } attemptId: 1 } id: 1 } state: C_COMPLETE diagnostics: "Exception from container-launch: \norg.apache.hadoop.util.Shell$ExitCodeException: \n\tat org.apache.hadoop.util.Shell.runCommand(Shell.java:458)\n\tat org.apache.hadoop.util.Shell.run(Shell.java:373)\n\tat org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)\n\tat org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)\n\tat org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)\n\tat org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)\n\tat java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:138)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)\n\tat java.lang.Thread.run(Thread.java:662)\n\n\n" exit_status: 1
2013-09-18 08:41:12,550 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:getNodeStatusAndUpdateContainersInContext(294)) – Removed completed container container_1379338026167_0125_01_000001
2013-09-18 08:41:14,556 INFO application.Application (ApplicationImpl.java:handle(430)) – Application application_1379338026167_0125 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2013-09-18 08:41:14,557 INFO containermanager.AuxServices (AuxServices.java:handle(161)) – Got event APPLICATION_STOP for appId application_1379338026167_0125
2013-09-18 08:41:14,557 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(368)) – Deleting absolute path : /hadoop/yarn/usercache/hdfs/appcache/application_1379338026167_0125
2013-09-18 08:41:14,559 INFO application.Application (ApplicationImpl.java:handle(430)) – Application application_1379338026167_0125 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2013-09-18 08:41:14,559 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(254)) – Application just finished : application_1379338026167_0125
2013-09-18 08:41:14,559 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainer(105)) – Starting aggregate log-file for app application_1379338026167_0125 at /app-logs/hdfs/logs/application_1379338026167_0125/ip-XXXXXXX.us-west-2.compute.internal_45454.tmp
2013-09-18 08:41:14,574 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainer(122)) – Uploading logs for container container_1379338026167_0125_01_000001. Current good log dirs are /hadoop/yarn
2013-09-18 08:41:14,575 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(377)) – Deleting path : /hadoop/yarn/application_1379338026167_0125
2013-09-18 08:41:14,626 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:doAppLogAggregation(182)) – Finished aggregate log-file for app application_1379338026167_0125
[/bash]