Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
•
1 like•307 views
In this report, we reviewed the remarkable AI developments in 2023 and made some predictions for new year! We will touch upon a wide range of topics -- technological advancements, startup activities, enterprise moves, investment trends, AI regulations, etc -- everything that we think are relevant for AI startups and VCs.
Recommended
More Related Content
What's hot
What's hot (11)
Similar to Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Similar to Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024) (20)
Recently uploaded
Recently uploaded (20)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
- 1. The State of AI January 24, 2024 Jenny Xiao Jay Zhao Reviewing 2023 and Predictions for 2024
- 2. Table of Contents Seven Trends in 202 Seven Predictions for 2024 2 CONFIDENTIAL
- 3. Jay Zhao Jenny Xiao Founder & General Partner Partner Former Partner @ T Fund, Former Board Member @ Marqeta & WorkBoard Ph.D. @ Columbia University, Former Researcher @ OpenAI About Us 3 CONFIDENTIAL Leonis Capital is a research-driven pre-seed fund for technical founders. Research-Driven We believe that having in-depth knowledge and a granular perspective drives superior returns and provides unique support to our companies at the earliest stage. First-Check Our research-backed conviction allows us to invest early. Founders love that we don’t require social proof and we get to optimize the fund’s return upside.
- 4. CONFIDENTIAL We do things differently... Here’s a snapshot of how much research we do...in 2023 alone. 400+ academic papers read 1,000+ companies mapped 241 pages of research slides produced 730h spent on delving into tech stacks 142 LLMs tracked and benchmarked 50% partner time spent on research 4
- 6. TheTransformercontinuestosee breakthroughsbutotherparadigmsareemerging. 2023Trend2: 6 CONFIDENTIAL FourBreakthroughsAreasinTransformersin2023 AIAgents MultimodalLLMs Efficiency Alignment Transformer-basedAIagents seesignificantbreakthroughs in2023withincreased reasoningcapabilitiesandmore matureframeworks. Examples:GenerativeAgents (StanfordandGoogle), Toolformer(Nvidia). Aligningmodelswithhuman intentionsbecomeeasieras researchersmovebeyond RLHF. Examples:DPO(Stanford), Zephyr(HuggingFace). MultimodalLLMsadvance quicklybutremainlargely undertheradarin2023. Examples:PaLM-E(Google). Methodsforimproving trainingefficiency significantly lowerthehardware requirementsfortrainingand fine-tuningLLMs. Examples:QLoRA (UWashington), FlashAttention-2(Stanford).
- 13. EnterprisestrybutstruggletobuildAI-first products. 2023Trend6: 13 CONFIDENTIAL Source:LeonisCapitalResearch. WhatPeopleThinkLLMProductsLookLike Lotsofenterpriseswanttobuild internalLLMappsorrolloutLLM products. !!Buildingaproduction- gradeLLMappisHARD WhatLLMProductsActuallyLookLike Finetunedataset LLM(asAPI) End-userapps Easy Easy Easy End-userapps Prompts, tasks& queries Users End-userapps Hard Hard Hard Personalization Prompts, tasks& queries Reinforcementlearningfrom humanfeedback(RLHF) Users LLMProvider Finetunedataset LLM(asAPI) Costoptimization Real-timeinference Easy Easy Easy Hard Hard/ Moderate Hard Moderate Moderate Moderate Moderate LLMProvider Company Dataprocessing pipelines Finetune dataset Training dataset Dataprocessing pipelines Vectordatabases Finetunedor trainedmodel
- 15. EnterprisestrybutstruggletobuildAI-first products. 2023Trend6: 15 CONFIDENTIAL AI-First TeamCompositionComparison TraditionalEnterpriseSaaS CTO Broaderrangeofroles,more people,moreakintoafactory. Fewerbutmorespecializedroles,more research-oriented,moreakintoalab. Backend team Quality assurance QAanalysts Product team Testers Productmanagers Frontendengineers UXdesigners Cloudarchitects Sitereliabilityengs Systemadmins Dataengineers Securityengineers Releaseengineers Middlewaredevs Databaseadmins Backendengineers DevOps team CTO Backend team Quality assurance AI QAanalysts Product team AI productmanagers AI productengineers ML infraengineers AI operationsengs AI platformengs Researchengineers AI securityengs Researchscientists AI/ML engineers DevOps team
- 17. AnewAIparadigmoutperformsthe Transformer. 2024Prediction1: 17 CONFIDENTIAL Reinforcement Learning(2015) Transformers& Diffusion(2017+) CNNs(2012) GANs(2014) OverdueforaParadigmShift? NextBigThing(?) Mamba? LNNs? Blending? Mostlyusedforimage classification,achieved muchbetterresultsthan otherdeeplearning architectures. Hyper-realisticimage generation,impressive “deepfake”humanfaces. RLmodelslearned strategicgameplay throughtrial-and-error andreward. Thebeginningof “generativeAI” -- AI is abletogeneratetext, image,audio,andvideo content.
- 20. Smallermodelsbecomemorepopularas companiesexploremoreusecases. 2024Prediction3: 20 CONFIDENTIAL Technique Examples Effects Quantization HardwareOptimization E.g.,QLoRA(UWashington2023), LoRA(Microsoft2021). E.g.,LLMinaFlash(Apple2023), FlashAttention-2(Stanford2023), FlashAttention(Stanford2022). E.g.,DPO(Stanford2023),PEFT (HKMU2023),RLHF(OpenAI 2017). E.g.,MiniLLM(Tsinghua2023), Distilling-Step-by-Step(Google 2023). Reducingthenumberofbits neededtorepresentinformation. Smallermodelsize Canrunmodelsonconsumer GPUswithminimalqualityloss. MethodstorunTransformers moreefficientlyonGPUs& more. MorecommonlyknownintheLLM worldasfine-tuning. Distillingknowledgefromalarge modeltoamuchsmallerone. TransferLearning KnowledgeDistillation Reducedmemoryusage. Fastertraining Allowsdeploymentonedge. Parameterefficiency. Improvedperformance Userfriendliness. Outperforminglargermodels Efficienttraining Reducedcomputationdemand.
- 25. VerticalAIstartupsdevelopmoremature applications. 2024Prediction6: 25 CONFIDENTIAL Vertical Startups+Verticals= GreenField+NewOpportunities BigTech+Verticals= HardtoSpecialize Startups+Horizontal= DirectCompetitionwithBigTech BigTech+Horizontal= Distributional+DataAdvantage Startups Big Tech WhereMost WinnersWillBe Hardtoworkwithverticalenterprises. Facesuspicionandanti-trustconcerns. Easiertoworkwithverticalenterprises. TheAImodel-datatradeisquitecommon. Lackexisting expertiseandinfrastructure. Mostdon’thaveexperienceinverticals. Movesfasterinhighly regulateddomains. Candothingsthatbig techwon’tdaredo.