• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            我要啦免费统计

            from http://docs.continuum.io/anaconda-cluster/examples/spark-caffe

            Deep Learning (Spark, Caffe, GPU)

            Description

            To demonstrate the capability of running a distributed job in PySpark using a GPU, this example uses a neural network library, Caffe. Below is a trivial example of using Caffe on a Spark cluster; although this is redundant, it demonstrates the capability of training neural networks with GPUs.

            For this example, we recommend the use of the AMI ami-2cbf3e44 and the instance type g2.2xlarge. An example profile (to be placed in ~/.acluster/profiles.d/gpu_profile.yaml) is shown below:

            name: gpu_profile
            node_id: ami-2cbf3e44 # Ubuntu 14.04 - IS HVM - Cuda 6.5
            user: ubuntu
            node_type: g2.2xlarge
            num_nodes: 3
            provider: aws
            plugins:
              - spark-yarn
              - notebook
            

            Download

            To execute this example, download the: spark-caffe.py example script or spark-caffe.ipynbexample notebook.

            Installation

            The Spark + YARN plugin can be installed on the cluster using the following command:

            $ acluster install spark-yarn
            

            Once the Spark + YARN plugin is installed, you can view the YARN UI in your browser using the following command:

            $ acluster open yarn
            

            Dependencies

            First, we need to bootstrap Caffe and its dependencies on all of the nodes. We provide a bash script that will install Caffe from source: bootstrap-caffe.sh. The following command can be used to upload the bootstrap-caffe.sh script to all of the nodes and execute it in parallel:

            $ acluster submit bootstrap-caffe.sh --all
            

            After a few minues, Caffe and its dependencies will be installed on the cluster nodes and the job can be started.

            Running the Job

            Here is the complete script to run the Spark + GPU with Caffe example in PySpark:

            # spark-caffe.py from pyspark import SparkConf from pyspark import SparkContext  conf = SparkConf() conf.setMaster('yarn-client') conf.setAppName('spark-caffe') sc = SparkContext(conf=conf)   def noop(x):     import socket     return socket.gethostname()  rdd = sc.parallelize(range(2), 2) hosts = rdd.map(noop).distinct().collect() print hosts   def caffe_process(x):     import os     os.environ['PATH'] = '/usr/local/cuda/bin' + ':' + os.environ['PATH']     os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda/lib64:/home/ubuntu/pombredanne-https-gitorious.org-mdb-mdb.git-9cc04f604f80/libraries/liblmdb'     import subprocess     proc = subprocess.Popen('cd /home/ubuntu/caffe && bash ./examples/mnist/train_lenet.sh', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)     out, err = proc.communicate()     return proc.returncode, out, err  rdd = sc.parallelize(range(2), 2) ret = rdd.map(caffe_process).distinct().collect() print ret 

            You can submit the script to the Spark cluster using the submit command.

            $ acluster submit spark-caffe.py 

            After the script completes, the trained Caffe model can be found at/home/ubuntu/caffe/examples/mnist/lenet_iter_10000.caffemodel on all of the compute nodes.

            posted on 2015-10-14 17:25 閱讀(3604) 評論(1)  編輯 收藏 引用 所屬分類: life關于人工智能的yy

            評論:
            # re: Deep Learning (Spark, Caffe, GPU) 2015-10-21 18:19 | 春秋十二月
            這是啥  回復  更多評論
              
            777米奇久久最新地址| 91久久精品视频| 无码专区久久综合久中文字幕| 久久久久久精品免费看SSS| 久久亚洲精精品中文字幕| 99久久99这里只有免费费精品| 66精品综合久久久久久久| 人妻丰满?V无码久久不卡| 久久美女网站免费| 香港aa三级久久三级老师2021国产三级精品三级在 | 久久这里只有精品18| 激情久久久久久久久久| 一本一本久久a久久综合精品蜜桃| 色综合久久最新中文字幕| 久久人与动人物a级毛片| 国产精品青草久久久久福利99 | 久久久免费观成人影院| 97久久超碰国产精品旧版| 国产亚洲美女精品久久久2020| 精品久久久久国产免费| 久久精品人人做人人爽电影| 97精品依人久久久大香线蕉97| 久久久久久久综合综合狠狠| 久久青草国产精品一区| 国产成人久久精品一区二区三区| 欧美日韩久久中文字幕| 国产精品美女久久久久av爽| 国产一级做a爰片久久毛片| 久久一日本道色综合久久| 亚洲中文字幕无码久久2017 | 7777久久亚洲中文字幕| 99久久精品日本一区二区免费| 色婷婷综合久久久久中文一区二区| 尹人香蕉久久99天天拍| 无码8090精品久久一区| 一本色道久久88综合日韩精品| 激情久久久久久久久久| 久久亚洲国产成人精品无码区| 久久久久久久久久久免费精品| 久久99精品久久久久久齐齐| 蜜臀久久99精品久久久久久|