This document discusses methods for calculating mesh saliency maps to guide large-scale mesh decimation for artist workflows. It proposes using the entropy of tropical angle of curvature at multiple scales as a robust saliency measure. An adaptive subsampling approach allows it to be calculated quickly while maintaining accuracy. Testing on models up to 150k vertices showed it outperforms previous methods in both speed and reliability. Feedback from game studios indicated it provides artists sufficient control over decimation results through adjustible saliency maps.
Mesh Saliency Detection Using Local Curvature Entropy for Large-Scale Model Decimation
1. Saliency Detection for Large-Scale
Mesh Decimation
Rafael Kuffner dos Anjos
Leeds, UK
Richard Andrew Roberts
VUW, NZ,
Benjamin Allen
VUW, NZ,
Joaquim Jorge
U Lisboa, PT
Ken Anjyo
OLM
2.
3.
4. 1- Faces: because we are humans
2- Regions with lots of detail, because we
are drawn to it
9. 1- Faces: because we are humans
2- Regions with lots of detail, because we
are drawn to it
How can one calculate this?
Mesh Saliency: what do we look at?
10. Look for details!
Look at mesh curvature
averages (Gaussian weighted)
at different scales, and
combine
Mesh Saliency: Lee et. al 2005
12. Look globally, in a Laplacian spectrum
Stochastic laplacians of
the mesh at different
scales.
“irregularity of the log-
Laplacian spectrum of a
3D mesh is highly related
to the saliency of the
mesh”
Mesh Saliency via Spectral Processing: Song et. Al 2014
14. Current Trend: Just let AI do it!
Mesh Saliency via Weakly Supervised
Classification-for-Saliency CNN: Song et. Al 2019
Let humans pick the
points, and then teach a
computer to pick the
points on its own!
15. Well, I have a fantasy game, and the
type of models I use have never been
seen by people….
16. Entropy of a random variable is the average level of
"information", "surprise", or "uncertainty" inherent in the
variable's possible outcomes.
Entropy on a 3D mesh
𝐻 𝑋 = −
𝑖=1
𝑛
𝑃 𝑥𝑖 log 𝑃(𝑥𝑖)
So if the information of a vertex or its vicinity is “surprising”,
we should keep it!
17. Local Curvature Entropy
Mesh Saliency Analysis via Local
Curvature Entropy: Limper et al. 2016
Instead of looking at the measurement
and its size, looking at how surprising
it is, at different scales.
18. +++ Concept of “surprise” will deal well with patterns
+++ Not Immediately local, can be used globally
+++ A lot faster
- - - Curvature might be unstable with weird topology
- - - Models from laser scan: noise is surprising
- - - Still not fast enough
Let’s try to solve this
21. The problem: Artist workflow.
Mesh decimation algorithms have one input: “how
much”.
Artists want to have control of how it is going to turn
out; where is the decimation going to be heavier.
Current workflow: manually paint weights on ZBrush,
or recreate everything from scratch.
22. Our Approach: provide control, fast.
New robust formulation for scanned,
modeled, sculpted meshes.
Parametrizable so artists have control.
Fast enough to allow multiple
iterations in a sequence.
Decimate using Saliency/Quadrics to
control where the detail goes.
23. Formulation
Entropy is a good
measure of saliency.
However, curvature is
not numerically stable
in common “production”
cases.
So we propose an
alternative measure.
24. Tropical Angle of Curvature 𝜃 𝑣
Simple measure of maximum
angle in a ring neighbourhood.
Fast calculate with dot product;
fixed range (0-1); good to
calculate entropy.
Also nice to detect creases,
which are challenging for
entropy alone.
25. Entropy of 𝜃 𝑣 , 𝐻(𝑣, 𝑟)
Instead of just the
entropy, of 𝜃 𝑣 , 𝛽 works
as a “contrast” parameter.
Adjusts the distribution
among discrete bins.
𝛽 can be optimized to
provide best distribution
possible (average “white”
saliency)
27. Noise reduction
In scanned models, “noise” is very
surprising.
So entropy will consider it salient.
minimize noise:
estimate if points fit to plane
[considering a noise level 𝜖, we
reduce its influence with a
smoothstep function.]
Quick way of fitting to a
plane
28.
29. Finally, Creases
We use tropical angle as a
crease detection tool, also
normalizing the values of
𝑅 𝑣 .
33. Going fast
This formulation makes sure results are accurate, reliable,
and controllable.
To make it fast we used “adaptive subsampling”.
34. First: good engineering.
Flat data structure with neighbors close in memory.
Parallel implementation as each sample is independent
Grouped Traversal, Priority queue, SIMD
This helped but did not do the trick.
Maybe if we fully moved to a GPU? But that was not the path
we chose. (next!)
35. How/why does it work?
Our estimate is multi-scale.
At large scales, not concerned about
“small scale details” that we might
miss.
At fine scales, need to sample enough
to catch those.
So, we use the same number of
samples “per neighbourhood”
36. This loses accuracy… Doesn’t it?
Not much.
Trick: choose a few samples to run the computation on but look at
all neighbors.
We guarantee that every computed saliency value is interpolated
from ground truth values, not subsampled.
and “Tropical Angle” is not heavily affected by tessellation.
Let’s look at results for the proof.
41. Reliability
Closest work to ours
(Limper 2016) failed
above 150k vertices.
Our method achieved
more reliability key to
industry adoption.
42. Machine Learning
Must work with
predecimated meshes.
We saw how badly that
affects accuracy.
Results from Song2019
are hit and miss for more
“fantastic” inputs”.
43. Human picked interest points
Are these good maps for the purpose of decimation?
These have other applications.
44. Decimation
Speaking of which, is this
good?
Yes! We still use quadrics,
but weight it with the
saliency information.
Pure saliency is too greedy.
Allows us to control where
the error goes.
49. Industry feedback:
Artists from SQEX and OLM got this tool to test with their
private models.
Everybody was able to find a suitable “saliency map” for their
monsters/humans/pokémons.
Most importantly, they were able to try MULTIPLE TIMES.
All feedback was on workflow/tool improvements. An overall
success!
50. Limitations
Animated models.
Better use of saliency map for decimation.
Interactive is perfect. For that we need to go faster.
Lighting/texture affects perceived saliency. Can that be
considered during the estimate?
Ask what the salient parts are in this sequence of slides. First two are pictures of human-like figures. SO it becomes easy
Similar here, has a “face”
What about this? Pointing up, detail as well
Image saliency studied this in the past. Looking at oriented responses in the image to find what “sticks out”
But sometimes it just doesn’t work.
When looking at meshes, we want to be able to generalize it!
Seminal work from Lee et. al introduces the concept. The authors define mesh saliency using a center-surround operator on Gaussian-weighted mean curvatures. This operator is applied in a scale-dependent manner, meaning that it captures geometric features at different scales. The authors argue that this definition of mesh saliency can capture what most people would classify as visually interesting regions on a mesh.
Moving to a more global approach:
When we talk about "stochastic Laplacians of the mesh at different scales", we are referring to a process where the Laplacian is computed in a stochastic (random) manner at different levels of detail or scales of the mesh. This could involve, for example, randomly sampling vertices or regions of the mesh at each scale and computing the Laplacian based on these samples. This allows for a multi-scale analysis of the mesh, capturing geometric features at different levels of detail.
The "log-Laplacian spectrum" of a 3D mesh refers to the spectrum (set of eigenvalues) of the logarithm of the Laplacian matrix. The spectrum of a matrix is a fundamental concept in linear algebra and provides important information about the properties of the matrix. In the context of a 3D mesh, the spectrum of the Laplacian can reveal important information about the geometric and topological properties of the mesh.
The statement "irregularity of the log-Laplacian spectrum of a 3D mesh is highly related to the saliency of the mesh" suggests that the degree of irregularity or variation in the log-Laplacian spectrum is a good indicator of the saliency (visually interesting or important features) of the mesh. In other words, regions of the mesh that cause significant irregularities in the log-Laplacian spectrum are likely to be regions that are visually salient or important. This could be because these regions represent significant changes or features in the geometry of the mesh, which are likely to attract visual attention.
Once a probability distribution has been defined, the entropy of the mesh can be computed using the formula for entropy from information theory:
H = - ∑ p(x) log(p(x))
where the sum is over all vertices or faces x in the mesh, and p(x) is the probability of x according to the defined distribution. This formula essentially adds up the amount of "surprise" or "information" associated with each vertex or face, with more probable vertices or faces contributing less to the sum, and less probable vertices or faces contributing more.
The resulting entropy value is a measure of the complexity or irregularity of the mesh. A mesh with high entropy has a lot of variation or complexity in its geometry, while a mesh with low entropy is more uniform or regular.
Local Curvature: For each point on the 3D mesh, compute the curvature. Curvature is a measure of how much a curve (in this case, the surface of the 3D mesh) deviates from being a straight line (or a plane, in 3D). High curvature corresponds to a high degree of bending or twisting at a point.
Probability Distribution: Create a probability distribution based on these curvature values. This could be done, for instance, by treating the curvature values as a discrete set of events and computing the frequency of each event.
Entropy Calculation: Compute the entropy of this distribution. In information theory, entropy is a measure of the uncertainty or randomness in a set of data. In this case, it measures the variability in the curvature values. The entropy is computed using the formula: H = - ∑ p(x) log(p(x)), where the sum is over all curvature values x, and p(x) is the probability of x according to the defined distribution.
The resulting Local Curvature Entropy is a measure of the complexity or irregularity of the 3D mesh, based on the distribution of curvature values. A high Local Curvature Entropy indicates a mesh with a wide variety of different curvatures (i.e., a complex or irregular shape), while a low Local Curvature Entropy indicates a mesh with mostly similar curvatures (i.e., a simple or regular shape).
Tell why “tropical”.
We derived Beta from a collection of models, so given a new model we can find the right beta.
Geodesic Neighborhood: For each vertex v on the 3D mesh, consider a geodesic neighborhood U(v,r) around the vertex. This neighborhood consists of all points x whose geodesic distance from v is less than r.
Probability Density Function: The tropical angle of curvature, θβ, is treated as a stochastic variable within the neighborhood U(v,r). The probability density function of θβ is approximated using an area-weighted histogram. This histogram is created by dividing the range of possible curvatures (from θmin to θmax) into n discrete intervals or "bins". The value Av for each bin is accumulated by considering the influence area of each vertex in the neighborhood.
Entropy Calculation: The local entropy H(v,r) is then calculated by summing the area-weighted contribution of each bin. A larger variety of θ(v) values (i.e., a wider range of curvatures) within the neighborhood would result in a higher entropy.
The parameter β is used to control the distribution of differences across the bins. A β value less than 1 will spread out the differences, while a β value greater than 1 will concentrate them. The text suggests that a suitable value for β can be determined automatically by setting the global entropy equal to a desired maximum histogram value.
Finally, the algorithm integrates a measure of "crease detection" to highlight local edges that may be difficult to parameterize with the entropy formulation. This can be useful in cases where it's important to preserve specific local features in the mesh, such as handcrafted patterns. The crease detection measure is weighted and added to the local entropy to produce the final saliency measure S(v).
Color mapping for all images in this paper is [0.0 − 1.0], cyan to red.
The method involves defining a level of "noise" ε, with the goal of removing artifacts from surface fluctuations smaller than this value. This is particularly effective on low entropy surfaces, such as flat planes. The approach approximates small neighbourhoods to a plane and identifies points within that area that have a variation within the noise level by measuring their distance to the estimated plane.
Example saliency maps of a model created from a scan (Thai Statue from Stanford Repository) where local noise is filtered out by using ε = 0.01% of the mesh area, as seen in the top of the model (A). (B) features a large value of ε in order not to apply this operation.
Changing globality parameter: top without noise removal, bottom with it.
Default parameters on a varied model set. Still works without tweaking.
Cutoff 2 days
Look at some of the “human interest points”, they have semantic meaning, not necessarily geometrical. They are suited for other applications. We could look into combining this in the future.
Comparison between Schelling points [26] probability distribution and our method. Schelling points tend to be discretized around individual areas of interest that may have contextual meaning, which is not always ideal for decimation (example, point at the center of the table). On the other hand, our approach works better at detecting creases and edges at different scales (e.g. see cup, table and airplane models). We argue that for mesh decimation ours is the best performing approach.
Top row: where the error is. Bottom: saliency maps.
Renders of Lucy. Left: After 99% decimation with our saliency map using a triangle mesh. Right: original with 14 million vertices. Bottom: Used saliency map and wire-frame output. Our method preserves salient features even under high compression levels, e.g., upper lip and feet details.
Left: saliency keeps Eyes, lips, hair, but subtle.
Renders of Lucy. Left: After 99% decimation with our saliency map using a triangle mesh. Right: original with 14 million vertices. Bottom: Used saliency map and wire-frame output. Our method preserves salient features even under high compression levels, e.g., upper lip and feet details
Left saliency: Keeps face and details. Hindu (right) models simplified at 99%. Both models have the same vertex count, with the saliency allowing more details to be preserved.