• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            我要啦免费统计

            from http://docs.continuum.io/anaconda-cluster/examples/spark-caffe

            Deep Learning (Spark, Caffe, GPU)

            Description

            To demonstrate the capability of running a distributed job in PySpark using a GPU, this example uses a neural network library, Caffe. Below is a trivial example of using Caffe on a Spark cluster; although this is redundant, it demonstrates the capability of training neural networks with GPUs.

            For this example, we recommend the use of the AMI ami-2cbf3e44 and the instance type g2.2xlarge. An example profile (to be placed in ~/.acluster/profiles.d/gpu_profile.yaml) is shown below:

            name: gpu_profile
            node_id: ami-2cbf3e44 # Ubuntu 14.04 - IS HVM - Cuda 6.5
            user: ubuntu
            node_type: g2.2xlarge
            num_nodes: 3
            provider: aws
            plugins:
              - spark-yarn
              - notebook
            

            Download

            To execute this example, download the: spark-caffe.py example script or spark-caffe.ipynbexample notebook.

            Installation

            The Spark + YARN plugin can be installed on the cluster using the following command:

            $ acluster install spark-yarn
            

            Once the Spark + YARN plugin is installed, you can view the YARN UI in your browser using the following command:

            $ acluster open yarn
            

            Dependencies

            First, we need to bootstrap Caffe and its dependencies on all of the nodes. We provide a bash script that will install Caffe from source: bootstrap-caffe.sh. The following command can be used to upload the bootstrap-caffe.sh script to all of the nodes and execute it in parallel:

            $ acluster submit bootstrap-caffe.sh --all
            

            After a few minues, Caffe and its dependencies will be installed on the cluster nodes and the job can be started.

            Running the Job

            Here is the complete script to run the Spark + GPU with Caffe example in PySpark:

            # spark-caffe.py from pyspark import SparkConf from pyspark import SparkContext  conf = SparkConf() conf.setMaster('yarn-client') conf.setAppName('spark-caffe') sc = SparkContext(conf=conf)   def noop(x):     import socket     return socket.gethostname()  rdd = sc.parallelize(range(2), 2) hosts = rdd.map(noop).distinct().collect() print hosts   def caffe_process(x):     import os     os.environ['PATH'] = '/usr/local/cuda/bin' + ':' + os.environ['PATH']     os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda/lib64:/home/ubuntu/pombredanne-https-gitorious.org-mdb-mdb.git-9cc04f604f80/libraries/liblmdb'     import subprocess     proc = subprocess.Popen('cd /home/ubuntu/caffe && bash ./examples/mnist/train_lenet.sh', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)     out, err = proc.communicate()     return proc.returncode, out, err  rdd = sc.parallelize(range(2), 2) ret = rdd.map(caffe_process).distinct().collect() print ret 

            You can submit the script to the Spark cluster using the submit command.

            $ acluster submit spark-caffe.py 

            After the script completes, the trained Caffe model can be found at/home/ubuntu/caffe/examples/mnist/lenet_iter_10000.caffemodel on all of the compute nodes.

            posted on 2015-10-14 17:25 閱讀(3604) 評論(1)  編輯 收藏 引用 所屬分類: life關于人工智能的yy

            評論:
            # re: Deep Learning (Spark, Caffe, GPU) 2015-10-21 18:19 | 春秋十二月
            這是啥  回復  更多評論
              
            久久久久亚洲av毛片大| 国产精品99久久精品| 香蕉99久久国产综合精品宅男自 | 久久99精品久久久久久野外| 狠狠精品久久久无码中文字幕| 亚洲精品NV久久久久久久久久 | 亚洲国产综合久久天堂| 久久精品国产99久久久| 蜜臀久久99精品久久久久久小说| 国产精品久久久久久久久| 久久久久国产精品三级网| 色综合久久久久久久久五月| 一本久久a久久精品综合香蕉| 亚洲色欲久久久综合网东京热| 久久99国产精品久久久| 久久久亚洲AV波多野结衣| 伊人色综合久久天天| 欧美大香线蕉线伊人久久| 深夜久久AAAAA级毛片免费看| AV无码久久久久不卡网站下载| 国产免费久久精品99re丫y| 亚洲国产精品久久久久久| 久久国产色AV免费看| 久久久精品人妻一区二区三区蜜桃| 久久久综合九色合综国产| 影音先锋女人AV鲁色资源网久久 | 精品综合久久久久久888蜜芽| 国产综合精品久久亚洲| 青青国产成人久久91网| 国产精品天天影视久久综合网| 中文字幕久久久久人妻| 综合网日日天干夜夜久久| 一本久久综合亚洲鲁鲁五月天亚洲欧美一区二区 | 日韩久久久久中文字幕人妻| 97久久精品人人澡人人爽| 99久久99这里只有免费的精品| 亚洲va久久久噜噜噜久久男同| 18岁日韩内射颜射午夜久久成人| 国产成人综合久久精品红| 久久综合久久美利坚合众国| 久久久亚洲裙底偷窥综合|