This document presents a framework for structure-preserving color transformations using Laplacian colormaps. (1) It represents images as graphs with Laplacian matrices that encode image structure. (2) It finds joint eigenbases of Laplacians to perform color transformations that minimally distort structure. (3) It applies this approach to applications like color-to-gray conversion, colorblind optimization, gamut mapping, and multispectral image fusion. The results demonstrate high-quality color transformations while preserving image structure.

1. Laplacian colormaps: a framework
for structure-preserving color transformations
Davide Eynard, Artiom Kovnatsky, Michael Bronstein
Institute of Computational Science, Faculty of Informatics
University of Lugano, Switzerland
Eurographics, 8 April 2014
This research was supported by the ERC Starting Grant No. 307047 (COMET).
1 / 40

2. 2 / 40

3. 3 / 40

4. 4 / 40

5. Color transformations
RGB source Luma
5 / 40

6. Color transformations
RGB source Luma
Standard color transformations may break image structure!
6 / 40

7. Color transformations
RGB source Luma Desired outcome
Standard color transformations may break image structure!
7 / 40

8. Image Laplacian
Input N × M image with d color
channels, column-stacked into an
NM × d matrix X
Represented as graph with K vertices
(e.g. superpixels) and weighted edges
8 / 40

9. Image Laplacian
Input N × M image with d color
channels, column-stacked into an
NM × d matrix X
Represented as graph with K vertices
(e.g. superpixels) and weighted edges
K × K adjacency matrix WX
wij = exp −
δ2
ij
2σ2
s
+
xki
− xkj
2
2
2σ2
r
K × K Laplacian
LX = DX−WX, DX = diag(
j=i
wij)
xki
xkj
wij
9 / 40

10. Laplacians = structure descriptors
UT
LXU = ΛX, VT
LYV = ΛY
X u4 u5 u6 u7
Y v4 v5 v6 v7
Similar structure ⇐⇒ similar Laplacian eigenvectors
10 / 40

11. Laplacians = structure descriptors
UT
LXU = ΛX, VT
LYV = ΛY
X u4 u5 u6 u7
Y v4 v5 v6 v7
Similar structure ⇐⇒ similar Laplacian eigenvectors
Ideally, two Laplacians are jointly diagonalizable (iff they
commute): there exists a joint eigenbasis ˆU = U = V
11 / 40

12. Laplacians = structure descriptors
X u2 u3 u4 u5
RGB source
12 / 40

13. Laplacians = structure descriptors
X u2 u3 u4 u5
RGB source
Y v2 v3 v4 v5
Luma (‘bad’ color conversion)
13 / 40

14. Laplacians = structure descriptors
X u2 u3 u4 u5
RGB source
Y v2 v3 v4 v5
Luma (‘bad’ color conversion)
Z t2 t3 t4 t5
‘Good’ color conversion
14 / 40

15. Laplacians = structure descriptors
X u2 u3 u4 u5 Clustering
RGB source
Y v2 v3 v4 v5 Clustering
Luma (‘bad’ color conversion)
Z t2 t3 t4 t5 Clustering
‘Good’ color conversion
15 / 40

16. Finding joint eigenbases
Joint approximate diagonalization
Find joint approximate
eigenbasis ˆU
min
ˆU
off( ˆU
T
LX
ˆU) + off( ˆU
T
LY
ˆU)
s.t. ˆU
T
ˆU = I
where off(A) = i=j a2
ij.
Cardoso 1995, Eynard et al. 2012, Kovnatsky et al. 2013
16 / 40

17. Finding joint eigenbases
Joint approximate diagonalization
Find joint approximate
eigenbasis ˆU
min
ˆU
off( ˆU
T
LX
ˆU) + off( ˆU
T
LY
ˆU)
s.t. ˆU
T
ˆU = I
where off(A) = i=j a2
ij.
Closest commuting Laplacians
Find closest commuting
pair ˜LX, ˜LY
min
˜LX,˜LY
˜LX − LX
2
F + ˜LY − LY
2
F
s.t. ˜LX
˜LY = ˜LY
˜LX
Since ˜LX and ˜LY commute, they
have a joint eigenbasis ˆU
Cardoso 1995, Eynard et al. 2012, Kovnatsky et al. 2013, Bronstein et al. 2013
17 / 40

18. Finding joint eigenbases
Joint approximate diagonalization
Find joint approximate
eigenbasis ˆU
min
ˆU
off( ˆU
T
LX
ˆU) + off( ˆU
T
LY
ˆU)
s.t. ˆU
T
ˆU = I
where off(A) = i=j a2
ij.
Closest commuting Laplacians
Find closest commuting
pair ˜LX, ˜LY
min
˜LX,˜LY
˜LX − LX
2
F + ˜LY − LY
2
F
s.t. ˜LX
˜LY = ˜LY
˜LX
Since ˜LX and ˜LY commute, they
have a joint eigenbasis ˆU
These two problems are equivalent!
(approx. joint diagonalizability ⇐⇒ approx. commutativity)
Cardoso 1995, Eynard et al. 2012, Kovnatsky et al. 2013, Bronstein et al. 2013
18 / 40

19. Laplacian colormaps
X
19 / 40

20. Laplacian colormaps
X
−→
Φθ
Y = Φθ(X)
Parametric colormap Φθ : RNM×d → RNM×d parametrized
by θ = (θ1, . . . , θn)
Global: each pixel x is transformed same way, y = Φθ(x)
Local: different transformations in q regions,
Φθ(X) =
q
i=1 wiΦθi
(X)
20 / 40

21. Laplacian colormaps
X
−→
Φθ
Y = Φθ(X)
LX LY
LX = DX − WX LΦθ(X) = DΦθ(X) − WΦθ(X)
Find an optimal parametric color transformation
min
θ∈Rn
LXLΦθ(X) − LΦθ(X)LX
2
F + regularization on θ
21 / 40

22. Color-to-gray conversion
Color mapping by a global color transformation of the form
Φθ(R, G, B) = θ1 + θ2Rθ3
+ θ4Gθ5
+ θ6Bθ7
Luma Col2Gray Rasche Decolorize Neumann Smith Lu Ours
2.18/-1.05 1.96/-0.10 1.43/-1.38 1.35/0.86 2.22/0.29 2.13/-0.29 1.47/0.82 1.19/1.15
ˇCad´ık 2008; Gooch et al. 2005; Rasche et al. 2005; Grundland, Dodgson 2007;
Neumann et al. 2007; Smith et al. 2008; Lu et al. 2012; Kuhn et al. 2008
22 / 40

23. Color-to-gray conversion
Color mapping by a global color transformation of the form
Φθ(R, G, B) = θ1 + θ2Rθ3
+ θ4Gθ5
+ θ6Bθ7
Luma Col2Gray Rasche Decolorize Neumann Smith Lu Ours
5.01/-0.55 3.42/-0.89 3.59/-0.48 3.44/1.41 5.44/-0.66 5.04/-0.19 2.90/0.50 1.28/0.86
9.27/-0.57 7.05/-0.53 7.20/-0.04 7.28/1.45 10.17/-1.05 9.13/-1.02 6.30/1.01 3.78/0.76
ˇCad´ık 2008; Gooch et al. 2005; Rasche et al. 2005; Grundland, Dodgson 2007;
Neumann et al. 2007; Smith et al. 2008; Lu et al. 2012; Kuhn et al. 2008
23 / 40

24. Color-to-gray conversion
Color mapping by a global color transformation of the form
Φθ(R, G, B) = θ1 + θ2Rθ3
+ θ4Gθ5
+ θ6Bθ7
Luma Col2Gray Rasche Decolorize Neumann Smith Lu Ours
0.97/0.27 1.24/-1.30 0.97/-0.08 1.02/0.61 1.66/-0.86 1.05/0.32 0.80/0.22 0.85/0.82
ˇCad´ık 2008; Gooch et al. 2005; Rasche et al. 2005; Grundland, Dodgson 2007;
Neumann et al. 2007; Smith et al. 2008; Lu et al. 2012; Kuhn et al. 2008
24 / 40

25. Color-to-gray conversion
Color mapping by a global color transformation of the form
Φθ(R, G, B) = θ1 + θ2Rθ3
+ θ4Gθ5
+ θ6Bθ7
Luma Col2Gray Rasche Decolorize Neumann Smith Lu Ours
RWMS 2.84 2.31 2.46 2.20 4.85 2.94 1.90 1.33
z-score -0.17 -0.31 -0.63 0.55 -0.53 -0.09 0.34 0.84
ˇCad´ık 2008; Gooch et al. 2005; Rasche et al. 2005; Grundland, Dodgson 2007;
Neumann et al. 2007; Smith et al. 2008; Lu et al. 2012; Kuhn et al. 2008
25 / 40

26. Computational complexity: Color-to-gray example
10-1
100
101
102
103
Time(sec)
#vertices253 641 1130 22946 91784 367136
0.597
RWMSerror
0.599
x10-3
Linear (n=3)
Non-linear (n=7)
Superpixels Scaling
Complexity O(K2)
Laplacian dimension K MN (realtime performance with
small K)
Optimization on θ is performed with small Laplacians. Then,
Φθ is applied on full image
Superpixels: Ren, Malik 2003
26 / 40

27. Color-blind image optimization
RGB source
X
Ψ
Seen by color-blind
Ψ(X)
Vi´enot et al. 1999, Kim et al. 2012
27 / 40

28. Color-blind image optimization
RGB source
X Φθ(X)
Ψ
Seen by color-blind
Φθ
(Φθ ◦ Ψ)(X)
28 / 40

29. Color-blind image optimization
RGB source
X Φθ(X)
Ψ
Seen by color-blind
Φθ
(Φθ ◦ Ψ)(X)
LXLΦθ(X)−LΦθ(X)LX
LXL(Φθ◦Ψ)(X)−L(Φθ◦Ψ)(X)LX
29 / 40

30. Color-blind image optimization: protanopia
RGB Lau
1.23
Optimized
0.50
Lau et al. 2011
30 / 40

31. Color-blind image optimization: tritanopia
RGB Lau
1.69
Optimized
0.53
Lau et al. 2011
31 / 40

32. Gamut mapping
Map image colors to a gamut G
(convex polytope)
min
θ∈Rn
LXLΦθ(X) − LΦθ(X)LX
2
F
+ regularization on θ
s.t. Φθ(X) ⊆ G
sRGB
G
32 / 40

33. Gamut mapping
Original Lau et al. Ours HPMINDE (clip)
Lau et al. 2011
33 / 40

34. RGB+NIR fusion
NIR RGB Lau et al. Ours
Lau et al. 2011
34 / 40

35. Multiple image fusion
Morning
Day
Evening
Night
Fusion
35 / 40

36. Summary
Framework
theoretically grounded
versatile
global/local
realtime
36 / 40

37. Summary
Framework
theoretically grounded
versatile
global/local
realtime
Applications
color-to-grayscale
color-blind optimization
gamut mapping
multispectral image fusion
37 / 40

38. Thank you!
38 / 40

39. Qualitative evaluation
Web survey
124 volunteers, 2884 pairwise evaluations
Thurstone’s law of comparative judgements → z-score
Consistent with ˇCad´ık’s results
39 / 40

40. Extension: local colormap
RGB Luma Lau et al.
Global Local Clusters
Φθ(X) = q
i=1 wiΦθi
(X)
Lau et al. 2011
40 / 40