• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            我要啦免费统计

            from http://docs.continuum.io/anaconda-cluster/examples/spark-caffe

            Deep Learning (Spark, Caffe, GPU)

            Description

            To demonstrate the capability of running a distributed job in PySpark using a GPU, this example uses a neural network library, Caffe. Below is a trivial example of using Caffe on a Spark cluster; although this is redundant, it demonstrates the capability of training neural networks with GPUs.

            For this example, we recommend the use of the AMI ami-2cbf3e44 and the instance type g2.2xlarge. An example profile (to be placed in ~/.acluster/profiles.d/gpu_profile.yaml) is shown below:

            name: gpu_profile
            node_id: ami-2cbf3e44 # Ubuntu 14.04 - IS HVM - Cuda 6.5
            user: ubuntu
            node_type: g2.2xlarge
            num_nodes: 3
            provider: aws
            plugins:
              - spark-yarn
              - notebook
            

            Download

            To execute this example, download the: spark-caffe.py example script or spark-caffe.ipynbexample notebook.

            Installation

            The Spark + YARN plugin can be installed on the cluster using the following command:

            $ acluster install spark-yarn
            

            Once the Spark + YARN plugin is installed, you can view the YARN UI in your browser using the following command:

            $ acluster open yarn
            

            Dependencies

            First, we need to bootstrap Caffe and its dependencies on all of the nodes. We provide a bash script that will install Caffe from source: bootstrap-caffe.sh. The following command can be used to upload the bootstrap-caffe.sh script to all of the nodes and execute it in parallel:

            $ acluster submit bootstrap-caffe.sh --all
            

            After a few minues, Caffe and its dependencies will be installed on the cluster nodes and the job can be started.

            Running the Job

            Here is the complete script to run the Spark + GPU with Caffe example in PySpark:

            # spark-caffe.py from pyspark import SparkConf from pyspark import SparkContext  conf = SparkConf() conf.setMaster('yarn-client') conf.setAppName('spark-caffe') sc = SparkContext(conf=conf)   def noop(x):     import socket     return socket.gethostname()  rdd = sc.parallelize(range(2), 2) hosts = rdd.map(noop).distinct().collect() print hosts   def caffe_process(x):     import os     os.environ['PATH'] = '/usr/local/cuda/bin' + ':' + os.environ['PATH']     os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda/lib64:/home/ubuntu/pombredanne-https-gitorious.org-mdb-mdb.git-9cc04f604f80/libraries/liblmdb'     import subprocess     proc = subprocess.Popen('cd /home/ubuntu/caffe && bash ./examples/mnist/train_lenet.sh', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)     out, err = proc.communicate()     return proc.returncode, out, err  rdd = sc.parallelize(range(2), 2) ret = rdd.map(caffe_process).distinct().collect() print ret 

            You can submit the script to the Spark cluster using the submit command.

            $ acluster submit spark-caffe.py 

            After the script completes, the trained Caffe model can be found at/home/ubuntu/caffe/examples/mnist/lenet_iter_10000.caffemodel on all of the compute nodes.

            posted on 2015-10-14 17:25 閱讀(3603) 評(píng)論(1)  編輯 收藏 引用 所屬分類: life關(guān)于人工智能的yy

            評(píng)論:
            # re: Deep Learning (Spark, Caffe, GPU) 2015-10-21 18:19 | 春秋十二月
            這是啥  回復(fù)  更多評(píng)論
              
            久久热这里只有精品在线观看| 久久精品无码一区二区WWW| 国产精品久久久久久久久免费| 免费国产99久久久香蕉| 99久久久久| 久久精品免费一区二区| 国产激情久久久久影院老熟女免费 | 久久婷婷午色综合夜啪| 蜜臀久久99精品久久久久久小说| 99热都是精品久久久久久| 久久国产色av免费看| 国产午夜电影久久| 久久久久人妻一区二区三区vr| 久久久精品视频免费观看| .精品久久久麻豆国产精品 | 狠狠色狠狠色综合久久| 国内精品久久久久久久亚洲| 亚洲精品国产字幕久久不卡| 久久久久久青草大香综合精品| 99久久99这里只有免费费精品| 亚洲精品乱码久久久久久蜜桃| 欧美久久综合性欧美| 久久国产高潮流白浆免费观看| 亚洲人成无码网站久久99热国产 | 看全色黄大色大片免费久久久 | 99久久精品国产一区二区| 人妻丰满?V无码久久不卡| 久久精品中文字幕第23页| 久久免费精品视频| 久久精品国产精品国产精品污| 日韩人妻无码精品久久久不卡 | 国产成人无码精品久久久性色 | 久久久久人妻一区精品性色av| av色综合久久天堂av色综合在| 天天做夜夜做久久做狠狠| 久久久这里有精品中文字幕| 久久精品成人一区二区三区| 久久久久亚洲?V成人无码| 青春久久| 久久久久久精品无码人妻| 亚洲va久久久久|