使用双引号
line.split("|")
时,它返回:
Array[String] = Array(C, h, r, i, s, t, o, p, h, e, r, |, J, a, n, " ", 1, 1, , " ", 2, 0, 1, 5, |, 5)
,这不是我需要的。
有人能帮我回答这个问题吗?谢谢!
scala> val feedbackmap = feedback.map(line=>line.split('|'))
feedbackmap: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[4] at map at <console>:29
scala> feedbackmap.first
19/04/10 14:15:55 INFO SparkContext: Starting job: first at <console>:32
19/04/10 14:15:55 INFO DAGScheduler: Got job 4 (first at <console>:32) with 1 output partitions
19/04/10 14:15:55 INFO DAGScheduler: Final stage: ResultStage 4 (first at <console>:32)
19/04/10 14:15:55 INFO DAGScheduler: Parents of final stage: List()
19/04/10 14:15:55 INFO DAGScheduler: Missing parents: List()
19/04/10 14:15:55 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[4] at map at <console>:29), which has no missing parents
19/04/10 14:15:55 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.4 KB, free 510.7 MB)
19/04/10 14:15:55 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2003.0 B, free 510.7 MB)
19/04/10 14:15:55 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:43371 (size: 2003.0 B, free: 511.1 MB)
19/04/10 14:15:55 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1008
19/04/10 14:15:55 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[4] at map at <console>:29)
19/04/10 14:15:55 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
19/04/10 14:15:55 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 5, localhost, partition 0,ANY, 2171 bytes)
19/04/10 14:15:55 INFO Executor: Running task 0.0 in stage 4.0 (TID 5)
19/04/10 14:15:55 INFO HadoopRDD: Input split: hdfs://nn01.itversity.com:8020/user/junyanxu/scenario_37/feedback.txt:0+58
19/04/10 14:15:55 INFO Executor: Finished task 0.0 in stage 4.0 (TID 5). 2173 bytes result sent to driver
19/04/10 14:15:55 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 5) in 7 ms on localhost (1/1)
19/04/10 14:15:55 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
19/04/10 14:15:55 INFO DAGScheduler: ResultStage 4 (first at <console>:32) finished in 0.007 s
19/04/10 14:15:55 INFO DAGScheduler: Job 4 finished: first at <console>:32, took 0.012483 s
19/04/10 14:15:55 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
res3: Array[String] = Array(Christopher, Jan 11, 2015, 5)
scala> 19/04/10 14:20:55 WARN SparkContext: Killing executors is only supported in coarse-grained mode
19/04/10 14:20:55 WARN ExecutorAllocationManager: Unable to reach the cluster manager to kill executor driver!
scala> val feedbackmap2 = feedback.map(line=>line.split("|"))
feedbackmap2: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[5] at map at <console>:29
scala> feedbackmap2.first
19/04/10 14:22:58 INFO SparkContext: Starting job: first at <console>:32
19/04/10 14:22:58 INFO DAGScheduler: Got job 5 (first at <console>:32) with 1 output partitions
19/04/10 14:22:58 INFO DAGScheduler: Final stage: ResultStage 5 (first at <console>:32)
19/04/10 14:22:58 INFO DAGScheduler: Parents of final stage: List()
19/04/10 14:22:58 INFO DAGScheduler: Missing parents: List()
19/04/10 14:22:58 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[5] at map at <console>:29), which has no missing parents
19/04/10 14:22:58 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.4 KB, free 510.7 MB)
19/04/10 14:22:58 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2003.0 B, free 510.7 MB)
19/04/10 14:22:58 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:43371 (size: 2003.0 B, free: 511.1 MB)
19/04/10 14:22:58 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1008
19/04/10 14:22:58 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[5] at map at <console>:29)
19/04/10 14:22:58 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
19/04/10 14:22:58 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 6, localhost, partition 0,ANY, 2171 bytes)
19/04/10 14:22:58 INFO Executor: Running task 0.0 in stage 5.0 (TID 6)
19/04/10 14:22:58 INFO HadoopRDD: Input split: hdfs://nn01.itversity.com:8020/user/junyanxu/scenario_37/feedback.txt:0+58