docker pull sequenceiq/spark:1.5.1

   
    
     http:
    
    //baidu
    
     .ku
    
    6
    
     .com
    
    /watch/
    
     8788485592428494013.
    
    html?page=videoMultiNeed
   
  

这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述

docker build –rm -t sequenceiq/spark:1.5.1

Running the image

1:
docker run -it -p 8088:8088 -p 8042:8042 -h sandbox sequenceiq/spark:1.5.1 bash
or
2:
docker run -d -h sandbox sequenceiq/spark:1.5.1 -d
这里写图片描述

   
    
     #
    
    
     # There is insufficient memory for the Java Runtime Environment to continue.
    
    
     # Native memory allocation (malloc) failed to allocate 358088704 bytes for committing reserved memory.
    
    
     # Possible reasons:
    
    
     #   The system is out of physical RAM or swap space
    
    
     #   In 32 bit mode, the process size limit was hit
    
    
     # Possible solutions:
    
    
     #   Reduce memory load on the system
    
    
     #   Increase physical memory or swap space
    
    
     #   Check if swap backing store is full
    
    
     #   Use 64 bit Java on a 64 bit OS
    
    
     #   Decrease Java heap size (-Xmx/-Xms)
    
    
     #   Decrease number of Java threads
    
    
     #   Decrease Java thread stack sizes (-Xss)
    
    
     #   Set larger code cache with -XX:ReservedCodeCacheSize=
    
    
     # This output file may be truncated or incomplete.
    
    
     #
    
    
     #  Out of Memory Error (os_linux.cpp:2726), pid=1851, tid=139714598016768
    
    
     #
    
    
     # JRE version:  (7.0_51-b13) (build )
    
    
     # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode linux-amd64 compressed oops)
    
    
     # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
    
    
     #
    
    ---------------
    
     T
    
    H
    
     R
    
    E
    
     A
    
    D  ---------------

Current thread (
    
     0x00007f11d0008800
    
    ):  JavaThread
    
     "Unknown thread"
    
    [_thread_in_vm, id=
    
     1884
    
    , stack(
    
     0x00007f11d6ec1000
    
    ,
    
     0x00007f11d6fc2000
    
    )]

Stack: [
    
     0x00007f11d6ec1000
    
    ,
    
     0x00007f11d6fc2000
    
    ],  sp=
    
     0x00007f11d6fc01f0
    
    ,  free space=
    
     1020
    
    k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+
    
     0x992f4a
    
    ]  VMError::report_and_die()+
    
     0x2ea
    
    V  [libjvm.so+
    
     0x4931ab
    
    ]  report_vm_out_of_memory(
    
     char
    
    
     const
    
    *,
    
     int
    
    , unsigned
    
     long
    
    ,
    
     char
    
    
     const
    
    *)+
    
     0x9b
    
    V  [libjvm.so+
    
     0x81338e
    
    ]  os::Linux::commit_memory_impl(
    
     char
    
    *, unsigned
    
     long
    
    ,
    
     bool
    
    )+
    
     0xfe
    
    V  [libjvm.so+
    
     0x81383f
    
    ]  os::Linux::commit_memory_impl(
    
     char
    
    *, unsigned
    
     long
    
    , unsigned
    
     long
    
    ,
    
     bool
    
    )+
    
     0x4f
    
    V  [libjvm.so+
    
     0x813a2c
    
    ]  os::pd_commit_memory(
    
     char
    
    *, unsigned
    
     long
    
    , unsigned
    
     long
    
    ,
    
     bool
    
    )+
    
     0xc
    
    V  [libjvm.so+
    
     0x80daea
    
    ]  os::commit_memory(
    
     char
    
    *, unsigned
    
     long
    
    , unsigned
    
     long
    
    ,
    
     bool
    
    )+
    
     0x2a
    
    V  [libjvm.so+
    
     0x87fcd3
    
    ]  PSVirtualSpace::expand_by(unsigned
    
     long
    
    )+
    
     0x53
    
    V  [libjvm.so+
    
     0x880c69
    
    ]  PSYoungGen::initialize_virtual_space(ReservedSpace, unsigned
    
     long
    
    )+
    
     0x89
    
    V  [libjvm.so+
    
     0x8815e0
    
    ]  PSYoungGen::initialize(ReservedSpace, unsigned
    
     long
    
    )+
    
     0x50
    
   
  

错误代码

   
    Java HotSpot(TM)
    
     64
    
    -Bit Server VM
    
     warning
    
    : INFO: os::commit_memory(
    
     0x00000000eaa80000
    
    ,
    
     358088704
    
    ,
    
     0
    
    ) failed;
    
     error
    
    =
    
     'Cannot allocate memory'
    
    (errno=
    
     12
    
    )
#
# There is insufficient
    
     memory
    
    
     for
    
    the Java Runtime Environment to
    
     continue
    
    .
# Native
    
     memory
    
    allocation (malloc) failed to allocate
    
     358088704
    
    bytes
    
     for
    
    committing reserved
    
     memory
    
    .
# An
    
     error
    
    report
    
     file
    
    with more information is saved as:
# /usr/local/spark-
    
     1.5
    
    
     .1
    
    -bin-hadoop2
    
     .6
    
    /bin/hs_err_pid1959.
    
     log
    
   
  

这里写图片描述

   
    bash-
    
     4.1
    
    
     # cat /proc/meminfo shows
    
    
     MemTotal:
    
    
     2040736
    
    kB
    
     MemFree:
    
    
     75800
    
    kB
    
     MemAvailable:
    
    
     108704
    
    kB
    
     Buffers:
    
    
     28624
    
    kB
    
     Cached:
    
    
     109588
    
    kB
    
     SwapCached:
    
    
     60224
    
    kB
    
     Active:
    
    
     866236
    
    kB
    
     Inactive:
    
    
     895504
    
    kB
Active(anon):
    
     793784
    
    kB
Inactive(anon):
    
     841408
    
    kB
Active(file):
    
     72452
    
    kB
Inactive(file):
    
     54096
    
    kB
    
     Unevictable:
    
    
     32
    
    kB
    
     Mlocked:
    
    
     32
    
    kB
    
     SwapTotal:
    
    
     1046524
    
    kB
    
     SwapFree:
    
    
     45888
    
    kB
    
     Dirty:
    
    
     256
    
    kB
    
     Writeback:
    
    
     0
    
    kB
    
     AnonPages:
    
    
     1563940
    
    kB
    
     Mapped:
    
    
     69960
    
    kB
    
     Shmem:
    
    
     11636
    
    kB
    
     Slab:
    
    
     98648
    
    kB
    
     SReclaimable:
    
    
     37960
    
    kB
    
     SUnreclaim:
    
    
     60688
    
    kB
    
     KernelStack:
    
    
     23056
    
    kB
    
     PageTables:
    
    
     34744
    
    kB
    
     NFS_Unstable:
    
    
     0
    
    kB
    
     Bounce:
    
    
     0
    
    kB
    
     WritebackTmp:
    
    
     0
    
    kB
    
     CommitLimit:
    
    
     2066892
    
    kB
    
     Committed_AS:
    
    
     7412308
    
    kB
    
     VmallocTotal:
    
    
     34359738367
    
    kB
    
     VmallocUsed:
    
    
     295844
    
    kB
    
     VmallocChunk:
    
    
     34359420732
    
    kB
    
     HardwareCorrupted:
    
    
     0
    
    kB
    
     AnonHugePages:
    
    
     679936
    
    kB
    
     CmaTotal:
    
    
     0
    
    kB
    
     CmaFree:
    
    
     0
    
    kB
    
     HugePages_Total:
    
    
     0
    
    
     HugePages_Free:
    
    
     0
    
    
     HugePages_Rsvd:
    
    
     0
    
    
     HugePages_Surp:
    
    
     0
    
    
     Hugepagesize:
    
    
     2048
    
    kB
    
     DirectMap4k:
    
    
     180096
    
    kB
    
     DirectMap2M:
    
    
     1916928
    
    kB
    
     cat:
    
    shows: No such file
    
     or
    
    directory
   
  

这里写图片描述
这里写图片描述

试运行截图记录

   
    ./bin/spark-submit examples/src/main/python/transitive_closure
    
     .py
    
   
  

这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述

Spark源码编译

这里写图片描述

   
    Error:(
    
     45
    
    ,
    
     66
    
    ) not found:
    
     
      type
     
     
      SparkFlumeProtocol
     
    
    
     val
    
    transactionTimeout: Int,
    
     val
    
    backOffInterval: Int) extends SparkFlumeProtocol
    
     with
    
    Logging {
   
  
   
    Error:(
    
     70
    
    ,
    
     39
    
    )
    
     not
    
    found: type EventBatch
  override
    
     
      def
     
     
      getEventBatch
     
     
      (n: Int)
     
     :
    
    EventBatch = {
   
  
   
    Error:(
    
     80
    
    ,
    
     22
    
    )
    
     not
    
    found: type EventBatch
    
     
      def
     
     
      getEventBatch
     
     :
    
    EventBatch = {
   
  

这里写图片描述
这里写图片描述