• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            我要啦免费统计

            from http://docs.continuum.io/anaconda-cluster/examples/spark-caffe

            Deep Learning (Spark, Caffe, GPU)

            Description

            To demonstrate the capability of running a distributed job in PySpark using a GPU, this example uses a neural network library, Caffe. Below is a trivial example of using Caffe on a Spark cluster; although this is redundant, it demonstrates the capability of training neural networks with GPUs.

            For this example, we recommend the use of the AMI ami-2cbf3e44 and the instance type g2.2xlarge. An example profile (to be placed in ~/.acluster/profiles.d/gpu_profile.yaml) is shown below:

            name: gpu_profile
            node_id: ami-2cbf3e44 # Ubuntu 14.04 - IS HVM - Cuda 6.5
            user: ubuntu
            node_type: g2.2xlarge
            num_nodes: 3
            provider: aws
            plugins:
              - spark-yarn
              - notebook
            

            Download

            To execute this example, download the: spark-caffe.py example script or spark-caffe.ipynbexample notebook.

            Installation

            The Spark + YARN plugin can be installed on the cluster using the following command:

            $ acluster install spark-yarn
            

            Once the Spark + YARN plugin is installed, you can view the YARN UI in your browser using the following command:

            $ acluster open yarn
            

            Dependencies

            First, we need to bootstrap Caffe and its dependencies on all of the nodes. We provide a bash script that will install Caffe from source: bootstrap-caffe.sh. The following command can be used to upload the bootstrap-caffe.sh script to all of the nodes and execute it in parallel:

            $ acluster submit bootstrap-caffe.sh --all
            

            After a few minues, Caffe and its dependencies will be installed on the cluster nodes and the job can be started.

            Running the Job

            Here is the complete script to run the Spark + GPU with Caffe example in PySpark:

            # spark-caffe.py from pyspark import SparkConf from pyspark import SparkContext  conf = SparkConf() conf.setMaster('yarn-client') conf.setAppName('spark-caffe') sc = SparkContext(conf=conf)   def noop(x):     import socket     return socket.gethostname()  rdd = sc.parallelize(range(2), 2) hosts = rdd.map(noop).distinct().collect() print hosts   def caffe_process(x):     import os     os.environ['PATH'] = '/usr/local/cuda/bin' + ':' + os.environ['PATH']     os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda/lib64:/home/ubuntu/pombredanne-https-gitorious.org-mdb-mdb.git-9cc04f604f80/libraries/liblmdb'     import subprocess     proc = subprocess.Popen('cd /home/ubuntu/caffe && bash ./examples/mnist/train_lenet.sh', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)     out, err = proc.communicate()     return proc.returncode, out, err  rdd = sc.parallelize(range(2), 2) ret = rdd.map(caffe_process).distinct().collect() print ret 

            You can submit the script to the Spark cluster using the submit command.

            $ acluster submit spark-caffe.py 

            After the script completes, the trained Caffe model can be found at/home/ubuntu/caffe/examples/mnist/lenet_iter_10000.caffemodel on all of the compute nodes.

            posted on 2015-10-14 17:25 閱讀(3606) 評論(1)  編輯 收藏 引用 所屬分類: life關于人工智能的yy

            評論:
            # re: Deep Learning (Spark, Caffe, GPU) 2015-10-21 18:19 | 春秋十二月
            這是啥  回復  更多評論
              
            久久精品亚洲男人的天堂| 狠狠色综合网站久久久久久久高清| 国产91久久精品一区二区| 久久99精品国产一区二区三区| 国产激情久久久久影院老熟女免费 | 狠狠综合久久AV一区二区三区| 丰满少妇高潮惨叫久久久| 日批日出水久久亚洲精品tv| 久久发布国产伦子伦精品| 亚洲精品99久久久久中文字幕| 久久久婷婷五月亚洲97号色| 久久97久久97精品免视看秋霞| 亚洲欧美伊人久久综合一区二区| 国产精品免费久久| 97久久香蕉国产线看观看| 精品久久久久久中文字幕大豆网| 久久露脸国产精品| 99久久国产热无码精品免费久久久久| 精品伊人久久大线蕉色首页| 久久久无码精品午夜| 日本精品久久久久中文字幕| av无码久久久久不卡免费网站| 久久精品国产亚洲AV蜜臀色欲 | 狠狠色丁香久久婷婷综合五月| 亚洲人AV永久一区二区三区久久 | 国产成人香蕉久久久久| 97精品伊人久久大香线蕉app| 中文字幕无码精品亚洲资源网久久| 久久精品国产色蜜蜜麻豆| 国产午夜精品久久久久九九电影| 久久精品成人免费看| 久久福利青草精品资源站免费| 2021精品国产综合久久| 国产精品久久一区二区三区| 久久亚洲国产中v天仙www| 亚洲综合精品香蕉久久网97| 亚洲国产精品久久久久婷婷老年| 99久久精品免费| 免费一级欧美大片久久网| 久久精品极品盛宴观看| 色狠狠久久AV五月综合|