On the Applicability of Convex Relaxations for Matching Symmetric Shapes

On the Applicability of Convex Relaxations for Matching Symmetric Shapes Nadav Dym Weizmann Institute of Science In collaboration with Haggai Maron and Yaron Lipman

Shape matching Finding isometries

Shape matching-finding isometries aa ii aa jj AA iiii = dd aa ii, aa jj BB kkkk = dd(bb kk, bb ll ) bb kk bb ll Goal: find mapping (permutation) σσ: aa 1,, aa nn bb 1,, bb nn such that AA iiii = BB σσ ii,σσ(jj)

Graph isomorphism Input: AA = AA TT, BB = BB TT Goal: find permutation (if exists) σσ such that AA iiii = BB σσ ii,σσ(jj) Goal: find permutation matrix PP Π nn such that AA = PP TT BBBB

Graph matching/quadratic assignment Input: AA = AA TT, BB = BB TT Output: PP Π nn such that AA PP TT BBBB: PP = aaaaaaaaaann PP Πnn AA PP TT BBBB FF PP = aaaaaaaaaann PP Πnn PPPP BBBB FF Graph matching

DS relaxation Π nn PP = aaaaaaaaaann PP Πnn PPPP BBBB FF NP Hard! conv Π nn SS = aaaaaaaaaann SS cccccccc Πnn SSSS BBBB FF

DS relaxation conv Π nn is the Birkhorff polytope: conv Π nn DDDD = SS SS 0, SS1 = 1, SS TT 1 = 1} DDDD relaxation: SS = aaaaaaaaaann SS DDDD SSSS BBBB FF GGMM DDDD (AA, BB)

DS relaxation-does it work? PP = aaaaaaaaaann PP Πnn PPPP BBBB FF Part I: exactness? SS PP SS = aaaaaaaaaann SS DDDD SSSS BBBB FF Part II: projection SS

Exactness affects projection efficiency Exact case Exact+small noise Very noisy SS SS S SS = aaaaaaaaaann SS DDDD SSSS BBBB FF

Π nn Part I: Exactness [Aflalo et al 2015], [Fiori and Sapiro 2015] Assume: (i) AA BB (ii) Unique isomorphism PP min AAAA SSSS FF = 0 SS DDDD PP Usually Usually not SS SS = aaaaaaaaaann SS DDDD SSSS BBBB FF

Problem: Unique solution assumption ψψ φφ multiple solutions ψψ, ψψ φφ symmetric

Symmetries of natural shapes Bilateral symmetry: [SCAPE, FAUST, TOSCA] [SHREC]

Exactness vs. convex exactness Exactness (asymmetric) PP SS Convex exactness (symmetric) PP 1 PP 2

Convex exactness-definition IIIIII AA, BB = PP Π nn AAAA = PPPP}, IIIIoo cccccccc AA, BB = SS DDDD AAAA = SSSS} GGMM DDDD (AA, BB) is convex exact if IIIIoo cccccccc (AA, BB) = cccccccc(iiiiii(aa, BB)) PP 1 PP 2

A convenient reduction (B=A) (easy) Lemma: GGMM DDDD (A, A) is convex exact For any BB s.t. BB AA, GGMM DDDD (A, B) is convex exact

Goal Almost surely Usually Usually not PP SS Almost surely Usually? Usually not? PP 1 PP 2

Measure for the space of asymmetric graphs Asymmetric graphs: {AA = AA TT AA has no non-trivial automorphisms} SS nn = AA AA = AA TT } Asymmetric graphs VV GG 1 μμ SS nn = LLLLLLLLLLLLLLLL VV GG 2 GG ii > {II nn } VV GG 3 VV GG = AA SS nn PPPP = AAAA, PP GG}

Measure for Graphs with prescribed symmetry group GG 0 Graphs with sym group GG 0 : Graphs whose automorphism group is GG 0 VV GG 0 = AA SS nn PPPP = AAAA, PP GG 0 }

Measure for Graphs with prescribed symmetry group GG 0 VV(GG 0 ) Graphs with sym group GG 0 VV GG 2 VV GG 1 μμ GG0 = LLLLLLesgue GG ii > GG 0 VV GG 3 VV GG = AA SS nn PPPP = AAAA, PP GG}

Convex exactness for reflective groups Theorem 1: If GG ZZ 2, then for μμ GG almost every AA, GGMM DDDD (AA, AA) is convex exact. Also true for GG = {II nn } PP 1 PP 2

In general Theorem 1 holds if GG is reflective (PP 2 = II nn, PP GG ). GG s action on the vertices has a full orbit. PP 0 = II 4 PP 1 PP 2 PP 4

?

General groups: 0-1 probability Theorem 2: For any GG Π nn, either (i) For μμ GG a.e. AA, GGMM DDDD (AA, AA) is convex exact. Or (ii) For all AA with sym group GG, GGMM DDDD (AA, AA) is not convex exact. Proof is constructive...

2 AA iiii = pp ii pp jj 2 AA iiii = pp ii pp jj 2 AA iiii = pp ii pp jj 1 GGMM DDDD (AA, AA) GGMM DDDD (AA, AA) GGMM DDDD (AA, AA)

General groups: 0-1 probability Theorem 2: For any GG Π nn, either (i) For μμ GG a.e. AA, GGMM DDDD (AA, AA) is convex exact. Or (ii) For all AA with sym group GG, GGMM DDDD (AA, AA) is not convex exact.

Summary- Part I convex exactness? almost everywhere almost everywhere

Part II: Where s my permutation? symmetric seeee oooo mmmmmmmmmmmmmmmmmmmm = SS DDDD AAAA = SSSS} Simplex algorithm

Part II: Where s my permutation? symmetric noise SS

DS relaxation- LL 2 projection S = aaaaaaaaaann SS DDDD SSSS BBBB FF SS PP LL2 = aaaaaaaaaann PP Π SS PP FF PP LL2

DS++: convex2concave projection SS PP LL2 PP ++

convex2concave projection [Zaslavskiy, Bach and Vert 2009] Observation: Convex energy EE 0 is equivalent over Π nn to concave energy EE TT. Concave energy: (i) Local/global minima are permutations! (ii) intractable Convex energy: (i) minima may not be permutations (ii) tractable! SS llllll SS SS gggggggggggg = PP

convex2concave projection convex concave EE tt0 EE tt1 EE tt2 EE TT SS kk = "aaaaaaaaaannn SS DDDD EE ttkk (SS) Warm start optimization from SS kk 1. SS 0 SS TT

DS++: convex2concave projection EE tt SS = SSSS BBBB FF 2 + tt(nn SS FF 2 ) convex concave EE tt0 EE tt1 EE tt2 EE ttff EE 0 = EE EE tt = EE over Π nn EE ttff strictly concave for t F 0 nn SS FF 2 = 0

Choosing [tt 0, tt FF ] EE tt SS = SSSS BBBB FF 2 + tt(nn SS FF 2 ) convex concave EE tt0 EE tt1 EE tt2 EE ttff Best choice of tt FF : tt FF = λλ mmmmmm tt FF = λλ mmmmmm over VV DDDD = SS SS1 = 0, SS TT 1 = 0} Best choice of tt 0 : (DS) tt 0 = 0 [Aflalo et al. 15] (DS+) tt 0 = λλ mmmmmm [Fogel et al. 13,15] (DS++) tt 0 = λλ mmmmmm over VV DDDD

Relaxation comparison

DS++ vs local minimization DS++ vs local minimization with 1000 different initializations:

Projection comparison Retrieval ratio LL 2 DS++ Noise (10 xx )

Symmetric, no noise EE tt SS = SSSS BBBB FF 2 + tt(nn SS FF 2 ) For tt > 0, PP ii are the only global minima! PP 1 PP 2

Symmetric, no noise Theorem 3: If DS(A,B) is convex exact, then (under some conditions) SS 1, SS 2,, SS TT = PP ii SS 0 SS TT PP 1 PP 2

Thank you! For more details see: Exact Recovery with Symmetries for the Doubly-Stochastic Relaxation. DS++: A Flexible, Scalable and Provably Tight Relaxation for Matching Problems. Acknowledgments: - European Research Council (ERC Starting Grant) - Israel Science Foundation