• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            我要啦免费统计

            from http://docs.continuum.io/anaconda-cluster/examples/spark-caffe

            Deep Learning (Spark, Caffe, GPU)

            Description

            To demonstrate the capability of running a distributed job in PySpark using a GPU, this example uses a neural network library, Caffe. Below is a trivial example of using Caffe on a Spark cluster; although this is redundant, it demonstrates the capability of training neural networks with GPUs.

            For this example, we recommend the use of the AMI ami-2cbf3e44 and the instance type g2.2xlarge. An example profile (to be placed in ~/.acluster/profiles.d/gpu_profile.yaml) is shown below:

            name: gpu_profile
            node_id: ami-2cbf3e44 # Ubuntu 14.04 - IS HVM - Cuda 6.5
            user: ubuntu
            node_type: g2.2xlarge
            num_nodes: 3
            provider: aws
            plugins:
              - spark-yarn
              - notebook
            

            Download

            To execute this example, download the: spark-caffe.py example script or spark-caffe.ipynbexample notebook.

            Installation

            The Spark + YARN plugin can be installed on the cluster using the following command:

            $ acluster install spark-yarn
            

            Once the Spark + YARN plugin is installed, you can view the YARN UI in your browser using the following command:

            $ acluster open yarn
            

            Dependencies

            First, we need to bootstrap Caffe and its dependencies on all of the nodes. We provide a bash script that will install Caffe from source: bootstrap-caffe.sh. The following command can be used to upload the bootstrap-caffe.sh script to all of the nodes and execute it in parallel:

            $ acluster submit bootstrap-caffe.sh --all
            

            After a few minues, Caffe and its dependencies will be installed on the cluster nodes and the job can be started.

            Running the Job

            Here is the complete script to run the Spark + GPU with Caffe example in PySpark:

            # spark-caffe.py from pyspark import SparkConf from pyspark import SparkContext  conf = SparkConf() conf.setMaster('yarn-client') conf.setAppName('spark-caffe') sc = SparkContext(conf=conf)   def noop(x):     import socket     return socket.gethostname()  rdd = sc.parallelize(range(2), 2) hosts = rdd.map(noop).distinct().collect() print hosts   def caffe_process(x):     import os     os.environ['PATH'] = '/usr/local/cuda/bin' + ':' + os.environ['PATH']     os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda/lib64:/home/ubuntu/pombredanne-https-gitorious.org-mdb-mdb.git-9cc04f604f80/libraries/liblmdb'     import subprocess     proc = subprocess.Popen('cd /home/ubuntu/caffe && bash ./examples/mnist/train_lenet.sh', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)     out, err = proc.communicate()     return proc.returncode, out, err  rdd = sc.parallelize(range(2), 2) ret = rdd.map(caffe_process).distinct().collect() print ret 

            You can submit the script to the Spark cluster using the submit command.

            $ acluster submit spark-caffe.py 

            After the script completes, the trained Caffe model can be found at/home/ubuntu/caffe/examples/mnist/lenet_iter_10000.caffemodel on all of the compute nodes.

            posted on 2015-10-14 17:25 閱讀(3604) 評論(1)  編輯 收藏 引用 所屬分類: life 、關于人工智能的yy

            評論:
            # re: Deep Learning (Spark, Caffe, GPU) 2015-10-21 18:19 | 春秋十二月
            這是啥  回復  更多評論
              
            一个色综合久久| 免费一级欧美大片久久网| 久久热这里只有精品在线观看| 亚洲综合久久夜AV | 久久久一本精品99久久精品88 | 久久精品亚洲乱码伦伦中文| 亚洲精品NV久久久久久久久久| 亚洲精品蜜桃久久久久久| 97精品国产97久久久久久免费| 久久久久女教师免费一区| 欧美黑人又粗又大久久久| 久久99热这里只有精品国产| 国产精品久久久久AV福利动漫| 久久青青草原精品国产不卡| 久久久噜噜噜久久熟女AA片| 久久亚洲2019中文字幕| 久久99中文字幕久久| 一本一本久久aa综合精品| 国产精品久久久久久久久久免费| 亚洲狠狠婷婷综合久久蜜芽| 色婷婷噜噜久久国产精品12p| 久久大香香蕉国产| 亚洲精品无码久久久久久| 国产精品久久久久久久久软件| 久久综合综合久久狠狠狠97色88| 午夜精品久久久久久久久| 性做久久久久久久久浪潮| 久久99精品久久久久久水蜜桃| 国产精品久久永久免费| 久久亚洲精精品中文字幕| 国产亚洲精久久久久久无码77777 国产亚洲精品久久久久秋霞 | 99久久国产热无码精品免费| 久久亚洲AV成人无码国产| 久久精品综合网| 亚洲综合日韩久久成人AV| 麻豆av久久av盛宴av| 久久天天躁狠狠躁夜夜avapp| 国产免费久久精品99re丫y| 久久精品无码一区二区WWW| 伊人色综合久久天天人手人婷| 热re99久久精品国99热|