deep learning in production cff 2017

1© Ari Kamlani 2017
skymind.ai | deeplearning4j.org | gitter.im/deeplearning4j
Ari Kamlani @akamlani
Skymind | Deep Learning Consultant
Cognitive Frameworks Festival (CFF) 06/09/2017

AGENDA
• Skymind Background
• DL4J Overview
• Environment Setup
• DL4J Component Breakdown
• Data Workflow
• Distributed Computed in DL4J
• Resource Goodies

TAKEAWAYS
• Developing your own network model in IntelliJ/Maven
• A High level understanding in migrating from other ML/DL frameworks
• Configuring the DL4J stack for your particular environment use case
• Scaling-Up vs Scaling-Out the network in DL4J

4© Ari Kamlani 2017 4© Ari Kamlani 2017
SKYMIND
+ Skymind Intro
+ Who we are

SKYMIND
Where you can find us…
Founded 2014
Funding $3.3M
Founders Adam Gibson, Chris Nicholson
Clients 12 enterprises
4,100+ community members
160,000+ DL4J downloads/mo.
Staff ~25 (Expected to Scale in 2017)

SKYMIND
Deep Learning for the Enterprise…
• RedHat for AI
• Custom Deep Learning Solutions the Enterprise (Java Focused)
• Focus is on a different User Base: Production vs Research
• Stack operates natively w/existing Big Data Infrastructure Architecture
• Build, Customize, and Maintain the complete Deep Learning Stack

HOW WE ARE USED
• Finance & Fraud (Financial Banks)
• Government (Nasa JPL)
• Telcom
• Retail (IBM, Amazon)
Customer Segments
Notable Uses Cases
• Anomaly Detection (Fraud Detection, Network Intrusion, Predictive Maintenance)
• Predictive Analytics (Demand Forecasting, Consumer Shopping Preferences)
• Computer Vision (Digital Stamp Recognition, OCR, Image Classification)
• Text Analytics (Chemical Compounds, FAA Part Document Versioning)
Partners

DL4J INTRO
+ Skymind DL4J Suite
+ High-Level Architecture

WHY JAVA
• Enterprises do not generally run Python in Production
• Python <-> Java Serialization is Expensive
• Modern Data Pipelines exist on the JVM (Provide a toolkit on the JVM)
• DL4J as First-Class Citizen to the JVM (Spark, Hadoop) on a cluster
Deep Learning for the Enterprise…

DL4J SUITE
• Apache Software Foundation (ASF) 2.0 Licensed
• Current Stable Release: 0.8.0 (Mar 2017)
• Java Client Interface w/underlying C++ Bindings
• Integrates well with other components of Big Data Ecosystem with ability to scale
The “Hadoop” of Deep Learning

SKYMIND STACK
Where our Stack fits in

SKYMIND CORE OPEN SOURCE

SKYMIND INTELLIGENCE LAYER (SKIL)
• Combines Open Source Components + Proprietary Vendor Integrations
• Provisioned Model Serving & Vendor Integration Deployment, Training of Scalable Models
• Vendor Integrations: Hadoop Distributions, Connectors, Chip/BLAS Integrations
• Model Serving ~ Tensorflow Serving provisioned for the Enterprise environment

ML WORKFLOW: PYTHON
Depending upon featurization/augmentation
Distributed/Tuning
VECTORIZEINGEST MODELING TUNING INFERENCE

ML WORKFLOW: DL4J
ETL
(DataVec)
Vectorize
(DataVec)
Modeling
(DL4J)
Evaluation
(Arbiter)
Execution Platform: e.g. Local, Cluster (Spark)
ND4J – Processor(CPU, GPU), Linear Algebra Runtime

DL4J SUITE

ENVIRONMENT
+ IntelliJ Setup
+ Configuration: pom.xml
+ Install Curated Examples
+ Install Snippets Repo

RESOURCES
• DL4J Examples: http://github.com/deeplearning4j/dl4j-examples
Repositories to clone

SETUP ENVIRONMENT
IntelliJ: From Scratch
• MAVEN-ARCHETYPE-QUICKSTART
Creates a basic template with pom.xml
http://nd4j.org/getstarted#starting-a-new-nd4j-project
• Afterwards: Enable Auto-Import
• Create Resource Directory: Mark as “Source Root”
IntelliJ: Import Project
• Follow DL4J Quickstart to import examples
https://deeplearning4j.org/quickstart#dl4j-
examples-in-a-few-easy-steps
• Import Project from External Model

MAVEN INTRO
Build System and Dependency Management
• Dependency Management via pom.xml file(s) – parent/child relationship
• Maven repository stored locally in ~/.m2
• In Scala ~ SBT (build.sbt)
General Format
<dependencies>
<dependency>
<groupId>${module.group.id}</groupId>
<artifactId>${module.group.name}</artifactId>
<version>${module.version}</version>
<scope></scope>
</dependency>
</dependencies>
<properties>

<datavec.version>${release.version}</datavec.version>
<nd4j.version>${release.version}</nd4j.version>
<dl4j.version>${release.version}</dl4j.version>
</properties>

DEPENDENCY MANAGEMENT
Maven Dependency Variants (pom.xml)
<dependency>
<groupId>org.datavec</groupId>
<artifactId>datavec-api</artifactId>
<version>${datavec.version}</version>
</dependency>
<dependency>
<artifactId>datavec-spark_${spark.scala.version}</artifactId>
<version>${datavec.spark.version}</version>
</dependency>
• org.{deeplearning4j, nd4j, datavec}
• org.apache.{hadoop, spark}
• org.scala-lang
Commonly Used: ${module.group.id}

DL4J Component
Breakdown
+ Component Migration/Responsibilities
+ Maven Intro
+ Workflow, Pipeline

DATAVEC
ETL for ML: Persistable Data Pipelines
• Data Preprocessing/Transformations: The Pandas of Enterprise Deep Learning
• Core Responsibilities: Data Ingestion, Normalization, Vectorization
• Extending into Transformation Process (e.g. Data Augmentation, Spark Transformations)
• Example Data Source Formats: JSON, CSV, Text, Image/Video, Audio
• “Canova” Module now integrated into DataVec (Per legacy context: <= Release 0.4)

DATAVEC ETL FOR ML
data.csv
Directory
Images
Transform
Process
Directory
Labeled Images
Path Label
Generator
Path Path
Label Generator
Record Reader
CSV
RecordReader
Image
RecordReader
RecordReaderDataSetIterator
ND4J Pre-Processor
Scaler/Standardizers
based on Data Format
Convert to INDArray

DATAVEC: ETL FOR ML

ND4J CORE
High Performance Tensor Library for Scientific Computing
• Multi-Platform Functionality including GPUs
• Numpy for the JVM, Linear Algebra & Signal Processing (Scipy) Framework
• Built for manipulation of Tensors, Matrices, INDArrays (NDArray)
• ND4J Indexing/Boolean Indexing (Slicing as in Numpy): via NDArrayIndex (INDArrayIndex)
• DataSet acts as a container for INDArray(s): e.g. features, labels
RecordReader recordReader = new CSVRecordReader(numLinesToSkip,delimiter);
recordReader.initialize(new FileSplit(new ClassPathResource("iris.txt").getFile()));
DataSetIterator iterator = new
RecordReaderDataSetIterator(recordReader,batchSize,labelIndex,numClasses);
DataSet allData = iterator.next();
allData.shuffle();
SplitTestAndTrain testAndTrain = allData.splitTestAndTrain(0.7);
DataSet trainingData = testAndTrain.getTrain();
DataSet testData = testAndTrain.getTest();

ND4J BACKENDS
• Function: Determines the flow of execution
• Swappable backends for linear-algebra libraries
and processors (more on this to come…)
• Default Config: execute on CPUs (native platform)
• Regardless of the Backend –
Same consistent ND4J API
ND4J
JavaCPP
libND4J
DL4J
HW Arch
Chip Lib
Native

NATIVE: IT’S NOT ALL JAVA
JavaCPP
• The Cython (C-Extensions for Python) for Java
• Performs as a bridge per Hardware Accelerator Libraries (e.g. cuDNN, MKL)
• Auto Generates Java Bindings from C++ (via Parsing) – avoid overhead of JNI
• For Memory Management off-heap (not JVM) – own GC for GPU CUDA/cuDNN
libND4J
• C++ Native Engine that powers ND4J for efficient compute operations on CPU/GPU
• Pluggable Hardware Architecture Libraries (NVIDIA cuDNN, Intel MKL)
• Drives the native performance (latency) required for computational operations
• Contains bindings for BLAS, LAPACK

DL4J TRAINING UI
• Browser Based (Default Port: 9000)
• Configure UI via Maven: -Dorg.deeplearning4j.ui.port=9001
• Real-Time Processing upon model.fit (vs TensorBoard)
• Remote UI Collection per Distributing Training (e.g. Spark)
Play-Based Web App
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-ui_${scala_binary_version}</artifactId>
<version>${dl4j.version}</version>
</dependency>
Maven Dependency (pom.xml)

ARBITER
Model Tuning NN Library: Search Space for Deep Learning
• Platform Agnostic Hyperparameter search per training and evaluating models
• Supports Grid and Randomized Search for MultiLayerNetwork, ComputationGraph
• UI(ArbiterUIServer) for evaluating search space
• Traditional ML (Python) ~ sklearn.model_selection{GridSearchCV, RandomizedSearchCV}
Maven Dependency (pom.xml)
<dependency>
<artifactId>arbiter-deeplearning4j</artifactId>
<version>${arbiter.version}</version>
</dependency>

WORKFLOW
+ Keras
+ Network Data Workflow

KERAS
• Keras Model Import: Released 0.7.1 (Dec 2016) – Support for Keras Pre v2.0
• Export Model Config/Weights from existing Keras model
• Keras as Frontend backed by JVM Stack
• Keras Integration (expected Q4 2017)
• DL4J Model Zoo

Keras Model Import (Trained Models from Keras into Dl4J)
Backends
Not related to ND4J Backends

TRANSFER LEARNING
Model Config Trained Model
Keras Model
Import
Train in DL4J
Model Serve/
Inference
Model Zoo
Keras
DL4J
Load Pretrained Models
from Keras
DL4J has its own Model Zoo too!
Refine Existing Models
Implement ‘Fine Tune’ Config
enforceTrainingConfig:
Training vs Inference
model.out(newSamples)
model.fit(trainData)

KERAS – DL4J NETWORK
Keras Model
Import
MultiLayerNetworkConfiguration ComputationGraphConfiguration
<dependency>
<artifactId>deeplearning4j-modelimport</artifactId>
<version>${dl4j.version}</version>
</dependency>
MultiLayerNetwork (conf) ComputationGraph (conf)
Equivalent to Keras Sequential Model:
Linear Stack of Layers
Equivalent to Keras Functional Model:
Complex Models, Non-Sequential, MIMO
Maven Dependency Variants (pom.xml)

SIMPLE DATA WORKFLOW
RecordReader
DataSetIterator
DataSet
NormalizerStandardize
shuffle
splitTestandTrain
MultiLayerNetworkConf
MultiLayerNetwork
• model.init
• model.setListeners
• model.fit(train)
• modelSerializer.writeModel(net)
• modelSerializer.restore*
• model.out(test)
• eval(testYhat, testLabels)
ModelSerializerModelSerializer
~ Vectorize Pipeline
ND4J Pre-Processor
Optional TP

DISTRIBUTED
COMPUTING
+ Applicability
+ Parallelism

EXECUTION
Selecting a Network
• Single CPU/GPU, Custom ASIC Variants (TPU)
• Networks: MultiLayerNetwork, ComputationGraph
• Single Node/Multi-Processor (e.g. Multi-GPU)
• Networks: ParallelWrapper
• Distributed across Nodes
• Networks: SparkDl4jMultiLayer, SparkComputationGraph

DATA PIPELINE
GPUs
• Optimized for Matrix Computations
• DL4J avoids additional Python <-> JVM Serialization
• Hybrid: Standardization, Featurization via initial layers as part of Model Architecture

I/O
INTENSIVE
COMPUTE
INTENSIVE
I/O
INTENSIVE
INGEST ETL FEATURIZATION MODELING
POST
TUNING
SERVING
LARGE CLUSTER
HIGH MEMORY
SMALL CLUSTER
SMALL MEMORY
GPUs for Training
Improve Training Time
Low RAM ~ 12-24 GB

PARALLELISM
• Delta in ingesting data & setting up training data, hyperparameter tuning
• Number of Parameters may grow to be large
• Update Large Numerical Multi-Dimensional Matrix
• Workloads either CPU vs GPU Intensive

Is Distributed worth the Cost
• Avoid Small & Shallow Networks
• Want Low Ratio of Network Transfer to Computation
• Good Candidates: Parameter Sharing (CNN, LSTM)

PARALLELISM
MODEL PARALLELISM
• Different partitions of the model assigned
to different nodes on a cluster – each
device operating on all data
• Allows for scaling to large Models, useful
when a model does not fit on single node
• Apply model compression to constrain the
size to improve network overhead
• Reference Google/Jeff Dean Paper
(Large Scaled Distributed Networks,
NIPS, 2012)

DATA PARALLELISM
• Apache Spark driven as Data Parallelism
• Model Replication w/devices having
different partitions of the data
• Results from each device are combined
MODEL & DATA PARALLELISM
• Examples: MXNet, Tensorflow

MULTI-GPU
To Enable Multi-GPU Access (pom.xml)
<properties>

<nd4j.backend>nd4j-cuda-8.0-platform</nd4j.backend>

<nd4j.backend>nd4j-native-platform</nd4j.backend>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native-platform</artifactId>
<version>${nd4j.version}</version>
</dependency>
<dependency>
<artifactId>nd4j-cuda-7.5-platform</artifactId>
</dependency>
<dependency>
<artifactId>nd4j-cuda-8.0-platform</artifactId>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<artifactId>${nd4j.backend}</artifactId>
</dependency>
</dependencies>
In Contrast to Tensorflow:
• pip install –-upgrade tensorflow-gpu
• Device Graph Scoping: tf.device(‘/gpu:<id>’)

MULTI-GPU
GPU Configuration
public static void main(String[] args) {
//Configure Cuda Environment
CudaEnvironment.getInstance().getConfiguration()
// key option enabled
.allowMultiGPU(true)
// allowing larger memory caches
.setMaximumDeviceCache(2L * 1024L * 1024L * 1024L)
// cross-device access is used for faster model averaging over pcie
.allowCrossDeviceAccess(true);
}
• Manage memory from Java:
via JavaCPP for own Memory Management of GPU (cuDNN)
• Per GPU allocation per job

FP HALF-PRECISION
FP16
• FP16 (2x less GPU RAM used) instead of FP32, FP64 to speed up Training
• Half Precision is good enough for approximate deep learning (not as sensitive)
• Smaller Precision, so tuning for loss may be more difficult
• Caveat: Not all variants of Linear Algebra Libraries are supported by FP16
• Powerful for memory-intensive computations
public static void main(String[] args) {
//Enable FP16
DataTypeUtil.setDTypeForContext(DataBuffer.Type.HALF);
}

RESOURCE MANAGERS
As a Scheduler
• Co-locate Resource where data is stored
• Co-locate compute with storage for fault tolerance
• Schedule as a Resource(available for job): Compute, Memory(RAM), Processor
• Mesos (GPU as schedulable quantity) vs YARN (only Compute, Memory)
• Execute distributed job across Nodes on pool of specified resources

APACHE SPARK
As a Scheduler
• Per Fast ETL & Fault tolerance at Worker Level
• Spark can use GPU as a resource (Mesos) as part of a job
• Spark 2.1.1 designed for use with Mesos >= 1.0.0
• DL4J uses Spark as a Data Access Layer (per Scheduler: does not manage data)
• Integrations: Cooperative Frameworks (Native: DL4J) vs Spark Bindings (TFoS)

APACHE SPARK
Spark Core
SQL Streaming MLlib GraphX DL4JBigDL
Partition
Partition
Partition
RDD
Use Spark RDD as Input to DataVec Transform Process
Natively on Spark

PARAMETER SERVER (PS)
Maintain State across workers (hold updated gradients)
• Separate Jobs: Workers (GPU, Low Memory), Parameter server (CPU, High Memory)
• Distributed Gradients from training batch fed into PS to produced new model parameters
• Centralized (DL4J) vs Decentralized (P2P: CoS) Parameter Synchronization Updates
• Synchronous (DL4J) vs Asynchronous Parameter Averaging (acts as Regularization)

SINGLE-NODE/MULTI-PROCESSOR
ParallelWrapper
• Single Node Parameter Averaging
• Via Model Duplication – Each Worker trains on same model
• Useful for testing Parameter Averaging on single node before going Distributed
Via Latency Hiding
• Asynchronous ETL to GPU in Background + Compute in Parallel using all cores
• Training Time = GPU ETL Time + Matrix Compute Time

DISTRIBUTED SPARK NETWORK
SparkDl4jMultiLayer(sc, conf, tm)
network
ParameterAveragingTrainingMaster
tm
MultiLayerNetworkConf confComputationGraphConf conf
SparkComputationGraph(sc, conf, tm)
network
network.fit(JavaRDD<DataSet> trainingData)Train Network
Evaluate Network network.evaluate(JavaRDD<DataSet> testData)
Note: DataSet is not a Spark DataSet!!!
network.init()
• Gather Statistics across DataSet
• Statistics per Columnar Basis
• Generate HTML or Text Output
network.setListeners(*)
SparkContext(scConf)
sc
Via SparkAnalyze

SPARK DEPENDENCIES
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<artifactId>datavec-spark_${spark.scala.version}</artifactId>
<version>${datavec.spark.version}</version>
</dependency>
<properties>
<datavec.spark.version>0.8.0_spark_2</datavec.spark.version>
<spark.major.version>2</spark.major.version>
<spark.version>2.1.1</spark.version>
<spark.scala.version>2.11</spark.scala.version>
</properties>
minimal pom.xml maven dependencies
e.g. For Spark Transformation Process (tp) ETL

RESOURCE
GOODIES
+ Useful Links: DL4J

DOCS
DL4J Documentation
§ http://nd4j.org/documentation
§ http://nd4j.org/doc
§ http://deeplearning4j.org/doc
§ http://deeplearning4j.org/datavecdoc
§ http://deeplearning4j.org/quickstart
§ http://deeplearning4j.org/etl-userguide
§ http://deeplearning4j.org/documentation
§ http://deeplearning4j.org/spark
§ http://deeplearning4j.org/gpu
§ http://deeplearning4j.org/model-import-keras
Pdfs
§ Model Parallelism: (Google 2012: Jeffrey Dean)
https://static.googleusercontent.com/media/research.google.com/en//archive/
large_deep_networks_nips2012.pdf

USEFUL EXAMPLES
§ dl4j-examples = http://github.com/deeplearning4j/dl4j-examples
§ {dl4j-examples}/tree/master/datavec-examples
§ {dl4j-examples}/tree/master/dl4j-cuda-specific-examples
§ {dl4j-examples}/tree/master/dl4j-spark-examples
Notable Getting Started DL4J Examples
Screencasts (any many more…)
§ Datavec Spark Transform:
http://github.com/SkymindIO/screencasts/tree/master/datavec_spark_transform
§ YouTube
http://www.youtube.com/watch?v=MLEMw2NxjxE

COMMUNITY
§ Github Repository
http://github.com/deeplearning4j
§ Gitter Channel
http://gitter.im/deeplearning4j/deeplearning4j
§ SF Meetup: Deep Learning in Production with Skymind
https://www.meetup.com/SF-Deep-Learning-in-Production/
Contribute – We’d love your help!
Issues
§ Extend Yarn to include GPU as a Resource:
http://issues.apache.org/jira/browse/YARN-5517

Thank You

CONTACT ME
If you want to contact Me:
Personal:
http://arikamlani.com/
akamlani@gmail.com
Twitter:
@akamlani
LinkedIn:
akamlani
SlideShare:
Ari Kamlani

deep learning in production cff 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to deep learning in production cff 2017

Similar to deep learning in production cff 2017 (20)

Recently uploaded

Recently uploaded (20)

deep learning in production cff 2017