• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            我要啦免费统计

            from http://docs.continuum.io/anaconda-cluster/examples/spark-caffe

            Deep Learning (Spark, Caffe, GPU)

            Description

            To demonstrate the capability of running a distributed job in PySpark using a GPU, this example uses a neural network library, Caffe. Below is a trivial example of using Caffe on a Spark cluster; although this is redundant, it demonstrates the capability of training neural networks with GPUs.

            For this example, we recommend the use of the AMI ami-2cbf3e44 and the instance type g2.2xlarge. An example profile (to be placed in ~/.acluster/profiles.d/gpu_profile.yaml) is shown below:

            name: gpu_profile
            node_id: ami-2cbf3e44 # Ubuntu 14.04 - IS HVM - Cuda 6.5
            user: ubuntu
            node_type: g2.2xlarge
            num_nodes: 3
            provider: aws
            plugins:
              - spark-yarn
              - notebook
            

            Download

            To execute this example, download the: spark-caffe.py example script or spark-caffe.ipynbexample notebook.

            Installation

            The Spark + YARN plugin can be installed on the cluster using the following command:

            $ acluster install spark-yarn
            

            Once the Spark + YARN plugin is installed, you can view the YARN UI in your browser using the following command:

            $ acluster open yarn
            

            Dependencies

            First, we need to bootstrap Caffe and its dependencies on all of the nodes. We provide a bash script that will install Caffe from source: bootstrap-caffe.sh. The following command can be used to upload the bootstrap-caffe.sh script to all of the nodes and execute it in parallel:

            $ acluster submit bootstrap-caffe.sh --all
            

            After a few minues, Caffe and its dependencies will be installed on the cluster nodes and the job can be started.

            Running the Job

            Here is the complete script to run the Spark + GPU with Caffe example in PySpark:

            # spark-caffe.py from pyspark import SparkConf from pyspark import SparkContext  conf = SparkConf() conf.setMaster('yarn-client') conf.setAppName('spark-caffe') sc = SparkContext(conf=conf)   def noop(x):     import socket     return socket.gethostname()  rdd = sc.parallelize(range(2), 2) hosts = rdd.map(noop).distinct().collect() print hosts   def caffe_process(x):     import os     os.environ['PATH'] = '/usr/local/cuda/bin' + ':' + os.environ['PATH']     os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda/lib64:/home/ubuntu/pombredanne-https-gitorious.org-mdb-mdb.git-9cc04f604f80/libraries/liblmdb'     import subprocess     proc = subprocess.Popen('cd /home/ubuntu/caffe && bash ./examples/mnist/train_lenet.sh', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)     out, err = proc.communicate()     return proc.returncode, out, err  rdd = sc.parallelize(range(2), 2) ret = rdd.map(caffe_process).distinct().collect() print ret 

            You can submit the script to the Spark cluster using the submit command.

            $ acluster submit spark-caffe.py 

            After the script completes, the trained Caffe model can be found at/home/ubuntu/caffe/examples/mnist/lenet_iter_10000.caffemodel on all of the compute nodes.

            posted on 2015-10-14 17:25 閱讀(3604) 評論(1)  編輯 收藏 引用 所屬分類: life 、關于人工智能的yy

            評論:
            # re: Deep Learning (Spark, Caffe, GPU) 2015-10-21 18:19 | 春秋十二月
            這是啥  回復  更多評論
              
            午夜不卡888久久| 日本久久久久久久久久| 久久久久久午夜成人影院| AA级片免费看视频久久| 久久精品视频一| 91久久精品视频| 日本欧美久久久久免费播放网| 爱做久久久久久| 久久亚洲私人国产精品| 香蕉久久永久视频| 色成年激情久久综合| 性欧美丰满熟妇XXXX性久久久| 国产亚洲成人久久| 国产亚洲欧美成人久久片| 久久精品国产亚洲AV蜜臀色欲| 国内精品久久久久久久影视麻豆| 亚洲AV无码久久| 久久久久久亚洲精品影院| 久久久精品久久久久久 | 国产精品99久久久久久宅男| 狠狠色狠狠色综合久久| 亚洲国产精品一区二区三区久久 | 亚洲人成网站999久久久综合| 精品久久久久久中文字幕| 人妻精品久久无码专区精东影业| 久久伊人色| 久久亚洲中文字幕精品一区| 久久国产V一级毛多内射| 国产99久久久久久免费看| 亚洲乱亚洲乱淫久久| 国产精品久久网| 日本精品久久久久中文字幕8| 99久久精品国产免看国产一区| 久久99国内精品自在现线| 久久久久久国产精品无码超碰| 久久久久国产精品人妻| 久久无码专区国产精品发布 | 久久天天躁狠狠躁夜夜2020 | 午夜不卡888久久| 久久久久国色AV免费观看| 色婷婷综合久久久久中文字幕|