第78回 Machine Learning 15minutes! Broadcast (2023/06/24) https://machine-learning15minutes.connpass.com/event/285147/ S/N Ratio > Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service (Machine Learning 15minutes! Broadcast #78) https://satonaoki.wordpress.com/2023/06/24/build-2023-updates-copilot-stack-azure-openai/

1. Microsoft Build 2023
Updates –
Copilot Stack and
Azure OpenAI Service
SATO Naoki (Neo)
Senior Software Engineer. Microsoft

2. +

3. World’s most
powerful
supercomputers
World’s most capable
foundation models, from
hosted to open source
World’s best
AI dev
infrastructure

4. Copilot

5. Announcing
Windows Copilot

6. Announcing
Copilot extensibility
and plugins
ChatGPT Bing Chat Microsoft Copilots Windows Copilot Your Copilots

7. Augment AI systems to access APIs
Retrieve useful information
Perform new computations
Safely act on the user’s behalf
Plugins

8. The digital world
Plugins
Copilot

9. Copilot
stack
AI infrastructure
Foundation models
AI orchestration
Orchestration
Copilots
Plugin extensibility
Apps

10. Copilot
stack
AI infrastructure
Foundation models
AI
safety
BYO models
Hosted fine-tuned foundation models
Hosted foundation models
Plugin extensibility
Plugin execution
Grounding
Metaprompt
Prompt & response filtering
Orchestration
Copilot frontend
UX

11. Foundation models
AI
safety
BYO models
Hosted fine-tuned foundation models
Hosted foundation models
Plugin extensibility
Copilot frontend
UX
Orchestration frameworks
Copilot stack
Plugin execution
Grounding
Orchestration
Metaprompt
Prompt & response filtering

12. Foundation models
AI
safety
Plugin extensibility
Plugin execution
Grounding
Orchestration
Copilot frontend
UX
Metaprompt
Prompt & response filtering
Prompt and Metaprompt
Copilot stack

13. Metaprompt
Foundation models
AI
safety
BYO models
Hosted fine-tuned foundation models
Hosted foundation models
Plugin extensibility
Plugin execution
Orchestration
UX
AI infrastructure
Vector databases • Web APIs • Plugins
Prompt & response filtering
Copilot stack
Retrieval Augmented Generation
(RAG)
Grounding

14. Foundation models
AI
safety
Grounding
Metaprompt
Prompt & response filtering
Orchestration
Copilot frontend
UX
Plugin execution
Plugin extensibility
Copilot stack
Plugins

15. AI
safety
Plugin execution
Grounding
Metaprompt
Prompt & response filtering
AI infrastructure
Foundation models
BYO models
Hosted fine-tuned foundation models
Hosted foundation models
Copilot stack
Foundation models and fine-tuning

16. AI infrastructure
Foundation models
AI
safety
BYO models
Hosted fine-tuned foundation models
Hosted foundation models
Plugin extensibility
Plugin execution
Grounding
Metaprompt
Prompt & response filtering
Orchestration
Copilot frontend
UX
Copilot
stack

17. Copilots
New development
pattern
Unique
architecture
Will be
everywhere

18. Azure AI
Applied AI Services
Bot Service Cognitive Search Form Recognizer Video Indexer Metrics Advisor Immersive Reader
Cognitive Services
Vision Speech Language Decision Azure OpenAI Service
Azure Machine Learning
Prepare & Preprocess Build, Train & Consume Deploy & Scale Manage & Monitor
AI Infrastructure

19. Announcing
Azure AI Studio
Build and
train your
own models
Ground Azure
OpenAI Service
and OSS models
using your data
Built-in
vector
indexing
Retrieval
augmented
generation
made easy
Create
prompt
workflows
AI safety
built-in

20. Generative AI Applications
Azure Machine Learning
Native OSS
Model Catalog
Prompt Eng/Eval
Prompt flow
Responsible AI
Azure Content Safety
High Scale Gen AI
App Deployment
Gen AI Model
Monitoring

21. Radically changing the art of possible
with Azure OpenAI Service
Large pretrained foundation AI models
custom-tunable with your parameters and your data
Summarization
Reasoning over data
Writing tools
Code generation
ChatGPT
The Era of Copilots
GPT-3 (GA) DALL•E 2 (preview) ChatGPT (GA) GPT-4 (GA)
Foundation of enterprise security, privacy and compliance

22. Enterprise innovation on Azure OpenAI Service

23. https://azure.microsoft.com/en-us/blog/mercedes-benz-enhances-drivers-experience-with-azure-openai-service/

24. Update
Bookmark ‘what's
new’ in Azure OpenAI
Service
Sign up for Azure
OpenAI Service
updates
Azure OpenAI Service
GPT-3 (GA) DALL·E 2 (preview) ChatGPT (GA) GPT-4 (GA)
Apply your
own data
Available in Preview
early June now
Plugins for Azure
OpenAI Service
Coming soon
Configurable
Content Filters
Available in Preview
early June now
Provisioned
Throughput
Limited Availability
early June

25. RAG: LLMs + your data
Retrieval Augmented Generation

26. Anatomy of a RAG app
App UX Orchestrator
Retriever over Knowledge Base
Query →
Knowledge
Prompt + Knowledge →
Response
Large Language Model
Build your own experience
UX, orchestration, calls to retriever and LLM
e.g., Copilots, in-app chat
Extend other app experiences
Plugins for retrieval, symbolic math,
app integration, etc.
e.g., plugins for OpenAI ChatGPT

27. Retrievers: Externalizing Knowledge
“Find the most relevant snippets
in a large data collection, using
unstructured input as query”
== search engine
App UX Orchestrator
Azure OpenAI
Azure Cognitive Search
Data Sources
(files, databases, etc.)
Query →
Knowledge
Prompt + Knowledge
→ Response
Azure Cognitive Search
 Azure’s complete retrieval solution
 Data ingestion, enterprise-grade
security, partitioning and replication
for scaling, support for 50+ written
languages, and more

28. Retrieving Using Semantic Similarity
Vector representations
(or embeddings)
 Learned such that “close” vectors
represent items with similar meaning
 May encode words, sentences, images,
audio, etc.
 Some map multiple media types into the same space
 Azure OpenAI embeddings API, OSS
embeddings (e.g., SBERT, CLIP)

29. Vector-based Retrieval
Encoding (vectorizing)
 Pre-process and encode
content during ingestion
 Encode queries during search/retrieval
Vector indexing
 Store and index lots of n-dimensional vectors
 Quickly retrieve K closest to a “query” vector
 Exhaustive search impractical in most cases
 Approximate nearest neighbor (ANN) search
Embedding
[0.023883354, 0.021508986,
0.044205155, 0.019588541,
0.031198505, …]

30. Vector Search in Azure Cognitive Search
New vector type for index fields
 Users indicate vector size, distance function, algorithm and algo-specific parameters
Pure Vector Search & Hybrid Search
 Filters, faceting, etc. all works with vectors
 Integrates with existing search indexes
 Existing data ingestion and augmentation machinery entirely applicable
Combines well with L2 re-ranker powered by Bing’s models
 Enables improved ranking for hybrid search scenarios
 L1: keywords + vector retrieval
 L2: Bing’s ranker refreshed with GPT-enhanced work
Enterprise-grade
 Scalability (partitioning, replication)
 Security: network isolation, managed identities, RBAC, etc.

31. Revolutionizing Indexing and
Retrieval for LLM-powered Apps
Power your retrieval-augmented generation applications
Images Audio Video Graphs Documents
• Use vector or hybrid search
• Use Azure OpenAI embeddings or bring your own
• Deeply integrate with Azure
• Scale with replication and partitioning
• Build generative AI apps and retrieval plugins
Sign up today https://aka.ms/VectorSearchSignUp

32. Azure OpenAI Service on your data

33. Chat Completions API
Versatile interface use for all scenarios—not just chat
Model adheres to instructions in “system” message
It sets the behavioral guidelines for the model, including responsible AI steering
Examples go into “User”, “Assistant” fields
import openai
openai.api_type = "azure”
response = openai.ChatCompletion.create(
engine ="gpt-4",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"},
{"role": "assistant", "content": "The LA Dodgers won the World Series in 2020."},
{"role": "user", "content": "Where was it played?"}
]
)
…

34. Using your data
Challenge I want to reason over my very
long documents, but the token
limits are not sufficient
Conventional solution Build a vector database
Retrieve relevant data and
construct prompt at run-time

35. Introducing:
Azure OpenAI Service on your data (Preview)
Data Sources
(search, files, databases, storage etc.)
Additional 3P Data Sources
(future capabilities)
Azure OpenAI Service
on your data
API & SDK
App or
Copilot agent

36. https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/introducing-azure-openai-service-on-your-data-in-public-preview/ba-p/3847000 https://www.youtube.com/watch?v=6SNfeVop4zM

38. Plugins
Azure OpenAI Service Plugins

39. Expanding potential
Challenges we wanted to address
Accurate translation
for wider range of
languages—improve
support for Asian and
African languages
Integrate vector
databases and cloud
data stores
Use up-to-date
information from
the web

40. Introducing:
Azure OpenAI Service Plugins (coming soon)
Build powerful AI Copilots with secure access to Microsoft services
Retrieve data with Azure
Cognitive Search
Translate >100 languages
with Azure Translator
Ground with recent info
with Bing Search
Extract structured data
from Azure SQL
Azure OpenAI
Plugins
• Securely access your data in
various data stores, vector
databases and the web
• Data path access controlled via
Azure AD and Managed Identities
• Admin roles to choose
what plugins to enable

41. Recap
Bookmark ‘what's
new’ in Azure OpenAI
Service
Sign up for Azure
OpenAI Service
updates
Azure OpenAI Service
GPT-3 (GA) DALL·E 2 (preview) ChatGPT (GA) GPT-4 (GA)
Apply your
own data
Available in Preview
early June
Plugins for Azure
OpenAI Service
Coming soon
Configurable
Content Filters
Available in Preview
early June
Provisioned
Throughput
Limited Availability
early June

42. Resources
 Microsoft Build (2023/05/24-26)
 https://build.microsoft.com/
 DEEP LEARNING LAB - [Recap] Microsoft Build 2023 最新アップデー
トAnalytics&AI (2023/06/08)
 https://dllab.connpass.com/event/284571/
 Microsoft Build 2023 Azure AI&ML 最新アップデート - Speaker Deck
 https://speakerdeck.com/shohei1029/microsoft-build-2023-azure-ai-and-ml-zui-xin-atupudeto
 Microsoft Build Japan (2023/06/27‐28)
 https://info.microsoft.com/JA-ADAI-CATALOG-FY23-06Jun-28-Microsoft-Build-Japan-Day2-
SREVM14500_Catalog-Display-Page.html

43. © Copyright Microsoft Corporation. All rights reserved.