Alluxio + Eckerson Webinar Sep. 12, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (Vice President of Research ) - Sridhar Venkatesh (SVP of Product) As enterprises race to roll out artificial intelligence, often overlooked are the infrastructure needs to support scalable ML model development and deployment. Efforts to effectively access and utilize GPUs often lead to extensive data engineering managing data copies or specialized storage, leading to out-of-control cloud and infrastructure costs. To address the challenges, enterprises need a new data access layer to connect compute engines to data stores wherever they reside in distributed environments. Join this webinar with Kevin Petrie, Eckerson Group VP of Research, and Sridhar Venkatesh, Alluxio SVP of Product, to explore tools, techniques, and best practices to remove data access bottlenecks and accelerate AI/ML model training. You will learn: - Modern requirements for AI/ML model training and data engineering - The challenges of GPU utilization in machine learning and the need for specialized hardware - How a new data access layer connects compute to data stores across environments - Best practices for optimizing ML training and guiding principles for success

1. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
Simplifying and Accelerating Data Access
for AI/ML Model Training
Kevin Petrie
Vice President of Research
Sridhar Venkatesh
SVP of Product

2. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
The Rise of Generative AI
After lots of training, language models generate strings of words
that become logical sentences and paragraphs
• A neural network whose nodes
share inputs and outputs
• Learns, summarizes, and
generates content
• Creates textual answers to
natural language questions
Source: The Economist
WHAT IS A LANGUAGE MODEL?

3. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
The Risk: Robots Get Things Wrong Too
Data teams must inspect, validate, and govern language model outputs
RISKS OF LANGUAGE MODELS
DATA QUALITY
Inaccuracies due to inaccurate/insufficient inputs, lack of context
EXPLAINABILITY
Vague/unknown sources or reasoning
PRIVACY
Exposure or theft due to user tracking
INTELLECTUAL PROPERTY
Liability for mishandled trademarks, copyrights, etc.
FAIRNESS
Perpetuation of bias in training data

4. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
Time to Get Domain Specific
Domain-specific, “small” language models reduce risk and boost productivity
by providing more governed and specialized outputs
• Enriched, detailed user
prompts
• Fine-tuned training on
enterprise data
• Augmented outputs; e.g.,
from multiple models
Small
Language
Model (SLM)
Large
Language
Model (LLM)
More
Governed
Generic Specialty
Less
Governed
ENTER THE SMALL
LANGUAGE MODEL

5. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
Small Language Models Will Drive the GenAI Boom
30% of data practitioners are building or training their own language models now. 20% more plan to do so*
*Source: Active LinkedIn survey of 55 respondents to date
“We believe in a world where
everyone is empowered to build
and train their own models,
imbued with their own opinions
and viewpoints.”
- Naveen Rao, Co-Founder and CEO,
MosaicML

6. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
“We hold these truths to be self-evident…”
TEXT
TOKENS We hold these …
VECTORS
truths to
[.45, 6.3, .99]
[7.6, .04, 19] [84, .13, 1.6]
VECTOR DB [.45, 6.3, .99] [7.6, .04, 19] [84, .13, 1.6]
LANGUAGE
MODEL
QUERY ONE QUERY TWO
1
2
3
4
5
Data teams must design and build new pipelines to feed their domain-specific data into language models
Data Processing for Language Models
Assemble unstructured text from various files
Convert words and punctuation marks to tokens
Use embeddings to convert tokens into numerical
vectors that describe their semantics
Load, organize, and index these vectors in a vector
database
Use a language model to search and query the
vectors while responding to real-time user prompts
NEW DATA PIPELINE

7. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
ON PREMISES | HYBRID | CLOUD | MULTI CLOUD
GENERATIVE AI EMBEDDED
CATALOG GOVERN
OBSERVE
INTEGRATE MASTER
STRUCTURED DATA
(DB TABLES)
SEMI STRUCTURED
(LOGS, CLICKSTREAMS, SENSORS…)
UNSTRUCTURED
(TEXT, IMAGES…)
CATALOG
INTEGRATE MASTER
ANALYTICS OPERATIONS
As companies embed generative AI into their workflows, they must manage
and process multi-structured data in a more holistic and efficient manner
The New Generative AI Data Stack

8. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
AI/ML Initiatives Need Fast and Simple Data Access
AI/ML initiatives require companies to balance, optimize, and secure workloads
across distributed datasets and compute resources
• Data access. View and process data wherever
it resides
• Performance. Retrieve data with low
latency/high throughput
• Portability. Run applications wherever suitable
compute resides
• Cost visibility. Monitor and control compute
cycles
• Multi tenancy. Isolate application compute to
safeguard performance
• Security. Restrict data access to minimize risk
of breaches
REQUIREMENTS

9. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
The Data Access Layer: Architecture
The data access layer continuously adjusts workloads, storage, and compute
• Namespace. Unified interface for
all data access
• APIs. Dynamic communication
between applications and storage
• Caching. Tier data by priority:
memory, SSDs, object store
• Metadata. Centralize descriptions
of data objects and resources
• Security. Authenticate users,
authorize access, log actions

10. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
Use Cases
AI initiatives have a range of use cases that require fast and simple data access
DATA CENTER CLOUD 1 CLOUD 2
ANALYTICS & AI IN A HYBRID ENVIRONMENT
ANALYTICS & AI ACROSS CLOUDS
WORKLOAD BURSTS
PROJECT EXPANSIONS
MIGRATIONS
COST OPTIMIZATION

11. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
Guiding Principles
Data leaders should evaluate where and how to implement a data access layer
to support generative AI initiatives
FIND THE BOTTLENECK
DECIDE WHETHER TO BUILD OR BUY
PLAN FOR GROWTH

12. © Eckerson Group 2021 Twitter: @eckersongroup www.eckerson.com
Questions?
I’m listening!

13. 13
Retooling the enterprise data infrastructure
Legacy data centers can’t keep up
High Performance
Computing
Specialized
Hardware
Varied
Workloads
We're seeing incredible orders to retool
the world's data centers… a 10-year
transition to basically recycle or reclaim
the world's data centers and build it out
as accelerated computing.
Jensen Huang
Nvidia CEO
“

14. Challenges as you try to scale
14
GPUs are this year’s
toilet paper.
Wall Street Journal
“
GPUs are
scarce
GPUs are
expensive
Low GPU
Utilization

15. Business Pressures Complex & Costly Solutions
GPUs are
scarce
GPUs are
expensive
Low GPU
Utilization
Faster model
development times
Increased
freshness
Higher accuracy
and traceability
Rapidly growing
datasets
Extensive data engineering
managing data copies
Specialized storage
Out of control cloud and
infra costs
15

16. 16
Alluxio Data Platform
High Performance data access, unified global view

17. 1.Faster Time-to-Market
50%
Hundreds of thousands of dollars saved annually
compared to previous deployment.
2-3X Model Training Performance Cost Reduction, Performance Boost
International B2C with a multi-cloud, cross-region AI platform, serving LLMs and training
models from object storage. They optimized their AI platform with Alluxio to speed data
delivery to training clusters and facilitate faster model deployment in latency sensitive
production use cases.
Models Deployed in Minutes vs Days
Faster model deployment times

18. 2. Higher GPU Utilization
“In a cloud environment, where GPU hardware is paid for as a function of time, you need
fast, performant, reliable, and cost effective data for your model training pipelines to keep
your GPU utilization close to 99%.”
20-30%
Average reported GPU utilization
based on direct access from remote
storage
GPU Utilization accessing commodity storage
GPU Utilization accessing Alluxio
Alluxio serves high throughput data to K8s training
workloads.
90
%
GPU utilization from Alluxio serving
data pulled from object storage. In
increase from 50% utilization via s3fs-
fuse.

19. 3. Reduction in Personnel
Increase in Productivity
Pre-Processed
Data
Data
Management
Pre-
Processed
Data
Training
Clusters
Data scientists
send requests to
AI platform
teams. Platform
teams set up
individual data
pipelines.
With Alluxio, data
scientists just
access their data.
Alluxio
consolidates many
pipelines into an
access layer.
Pipeline or
Scheduler
Training
Clusters

20. 20
4. Reduction in Infrastructure Spend
Alluxio optimizes data platforms to increase efficiency
Data Engineering
Pipelines
Data workflows improved by on-
demand access from Alluxio cache
S3 Egress and API
Fees
Fees significantly reduced via
granular caching and data
reuse
High Performance
Computing
Replaceable with low-cost hardware
at comparable performance
Reduced or Eliminated
Network Congestion Network congestion reduced by
serving files locally

21. 5. Cloud Vendor Leverage
Multi-cloud strategies with cost-effective benefits
Respond to Limited GPU Availability
Demand for GPUs has exploded
Organizations use Alluxio to supply high performance data access
to remote GPU clusters wherever they find capacity.
Increase Cloud Agility
Competing CSPs may provide attractive discounts
Alluxio empowers organizations to capitalize on hardware discounts
or cost-effective storage in real-time. Users access data wherever it
resides.
Avoid Vendor Lock-In
Negotiate with CSPs from a stronger position
Single cloud deployments are convenient, but that may become an
obstacle in negotiations. Alluxio facilitates hybrid and multi-cloud.

22. Twitter.com/alluxio
Linkedin.com/alluxio
Website
www.alluxio.io
Slack
http://slackin.alluxio.io/
@
Social Media
Q&A