Applying weightwatcher to LLMs Deck for the LLM Avalanche 2023 conference

1. My most viral (& controversial) LinkedIn post
w|w
(TM)
AI is hard; LLMs can over
fi
t
Can GPT4 actually ‘ace’ all MIT courses ?
?
Or did they over
fi
t a custom LLM to the past exams ?

2. weight|watcher
Data Free Diagnostics for Deep Learning
(TM)
charles@calculationconsulting.com

3. (TM)
3
Open source tool:
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

4. (TM)
4
Evaluating Base Models: BLOOM
Smaller
alpha
is better
Signatures
of
Emergence
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)
Layer quality metric—alpha

5. (TM)
5
Comparing LLMs: Falcon vs Llama
Falcon Llama
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

6. (TM)
6
Comparing LLMs: Correlation Flow
Falcon Llama
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

7. (TM)
7
Fined-Tuned LLMs: Only the deltas
Vicuna Dromedary
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

8. w|w
(TM)
(TM)
8
LLM Base Models: Truthfulness
TruthfulQA
metric
weightwatcher alpha
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

9. w|w
(TM)
(TM)
9
LLM Base Models: Training Quality
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

10. WeightWatcher: why Power Law
fi
ts ?
(TM)
10
Spiking (i.e real) neurons exhibit power law behavior
weightwatcher supports several PL
fi
ts
from experimental neuroscience
plus totally new shape metrics
we have invented (and published)
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

11. WeightWatcher: why Power Law
fi
ts ?
(TM)
11
Spiking (i.e real) neurons exhibit (truncated) power law behavior
The Critical Brain Hypothesis
Evidence of Self-Organized Criticality (SOC)
Per Bak (How Nature Works)
As neural systems become more complex
they exhibit power law behavior
and then truncated power law behavior
We see exactly this behavior in DNNs
and it is predictive of learning capacity
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

12. WeightWatcher: open-source, open-science
(TM)
12
We are looking for early adopters and collaborators
https://github.com/CalculatedContent/WeightWatcher
We have a Discord channel to support the tool
100K+ downloads; 1000 stars
https://discord.com/invite/uVVsEAcfyF
w|w
(TM)
weight|watcher.ai data-free model monitoring
(TM)

13. (TM)
c|c
(TM)
c | c
charles@calculationconsulting.com