UpBit:'Scalable'In0Memory' Updatable'Bitmap'Indexing

Transcription

1 UpBit:'Scalable'InMemory' Updatable'Bitmap'Indexing Manos&Athanassoulis Harvard&University Talk&at&CS65,&September&26 th,&26

2 Indexing'for'Analytical'Workloads Column&A 3 A= A=2 A=3 Specialized&indexing Compact&representation&of&query&result 2 3 Query&result&is&readily&available 2 Bitvectors Can&leverage&fast&Boolean&operators 3 2 Bitwise&AND/OR/NOT&faster&than&looping&over&meta&data 2

3 Bitmap'Indexing'Limitations Index&Size Column&A 3 A= A=2 A=3 SpaceQinefficient&for&large&domains 2 3 Addressed&by&bitvector&encoding/compression 2 core'idea:&run$length*encoding*in&prior&work 3 2 but$ Updating&encoded&bitvectors&is&very inefficient 3

4 4 3&zeros ending&pattern Update? &zeros ending&pattern flip*bit re$encode encode decode

5 Goal Bitmap&Indexing&with&efficient&Reads&&&Updates 5

6 Prior%Work:'Bitmap'Indexing'and'Deletes Update&Conscious&Bitmaps&(UCB),&SSDBM&27 A= A=2 A=3 EB efficient&deletes&by&invalidation existence&bitvector&(eb) 6

7 Prior%Work:'Bitmap'Indexing'and'Deletes efficient&deletes&by&invalidation existence&bitvector&(eb) reads? bitwise&and&with&eb updates? deleteqthenqappend 7 A= A=2 A=3 EB EB A=2 Update&Conscious&Bitmaps&(UCB),&SSDBM&27

8 Prior%Work:'Limitations Latency (ms) Bitwise AND with EB Decode VB Update and Encode EB Decode EB n=m&tuples,&d=&domain&values,&5%&updates&/&5%&reads U R U R U R U R U R UCB st UCB K UCB K UCB 2K UCB 4K read&cost&increases&with&#updates why? bitwise&and&with&eb&is&the&bottleneck update&eb&is&costly&for&>>&#updates UCB&performance&does&not&scale&with&#updates single&auxiliary&bitvector& 8 repetitive&bitwise&operations

9 Bitmap'Indexing'for'Reads'&'Updates distribute&update&cost efficient&random&accesses&in&compressed&bitvectors queryqdriven&requse&results&of&bitwise&operations 9

10 Design'Element':'update'bitvectors A= UB one&per&value&of&the&domain initialized&to&s the&current&value&is&the&xor every&update&flips&a&bit&on&ub

11 Design'Element':'update'bitvectors A= UB one&per&value&of&the&domain initialized&to&s the&current&value&is&the&xor every&update&flips&a&bit&on&ub &distribute&the&update&burden

12 Updating'UpBit 2 &row&2&to& A= A=2 A=3 UB UB UB

13 Updating'UpBit 3 &row&2&to&. find&old&value&of&row&2&(a=2) A= A=2 A=3 UB UB UB

14 Updating'UpBit 4 &row&2&to&. find&old&value&of&row&2&(a=2) A= A=2 A=3 UB UB UB

15 Updating'UpBit 5 &row&2&to&. find&old&value&of&row&2&(a=2) 2. flip&bit&of&row&2&of&ub&of&a=2 A= A=2 A=3 UB UB UB

16 Updating'UpBit 6 &row&2&to&. find&old&value&of&row&2&(a=2) 2. flip&bit&of&row&2&of&ub&of&a=2 3. flip&bit&of&row&2&of&ub&of&a= A= A=2 A=3 UB UB UB can&we&speed&up&step&?

17 Design'Element'2:'fence'pointers efficient&access&of&compressed&bitvectors fence'pointers row&3 row&6 7

18 Updating'UpBit 8 &row&2&to&. find&old&value&of&row&2&(a=2) A= A=2 A=3 UB UB UB

19 Updating'UpBit (with'fence'pointers) 9 &row&2&to&. find&old&value&of&row&2&(a=2) using&fence&pointers A= A=2 A=3 UB UB UB row&2

20 Querying 2

21 Querying'UpBit 2 &A&=&2 Return&the&XOR&of&A=2&and&UB A= A=2 A=3 UB UB UB

22 Querying'UpBit &A&=&2 Return&the&XOR&of&A=2&and&UB A=2 UB A=2 can&we&requse&the&result? 22

23 Design'Element'3:'querydriven'merging A=2 UB A=2 UB maintain&high&compressibility&of&ub query2driven'merging on&query& A=2 23

24 UpBit supports'very'efficient'updates n=m&tuples,&d=&domain&values k&queries&(varying&%&of&updates) Update Latency (ms) In-place UCB UpBit Read Latency (ms) Read-optimized UCB UpBit % update 5% update % update updates:&5q29x&faster&than&ucb 5Q5x&faster&than&inQplace % update 5% update % update only&8%&read&overhead&over&optimal 3x&faster&reads&than&UCB 24

25 UpBit offers'robust'reads Update Latency (ms) Read Latency (ms) In-place UCB UpBit % update 5% update % update updates:&5q29x&faster&than&ucb 5Q5x&faster&than&inQplace Read Latency (ms) # updates (thousands) 5 n=m&tuples,&d=&domain&values 5%/5%&update/read&queries UCB UpBit Ideal In-place UCB UpBit % update 5% update % update only&8%&read&overhead&over&optimal 3x&faster&reads&than&UCB 25

26 More'details' ' Tuning: how&frequent&to&merge&ub&to&the&index? Tuning:'what&is&the&optimal&granularity&of&fence&pointers? Optimizations: multiqthreaded&reads&and&updates Performance: full&query&analysis&(scientific&data&and&tpch)

27 Tuning when&to&merge?&(during&reads) inqplace&updates?&(fence&pointers) how&frequently&to&merge? how&frequent&fence&pointers? Update Latency (ms) Update Read Merging threshold merge&back&every&q2&updates Read Latency (ms) Latency (ms) Read Update Memory length = 6 length = 5 length = 4 length = 3 length = 2 length = length = no FP length = fence&pointers&every& 4 Q 5& values Space Overhead (ratio)

28 Memory'Consumption Space Overhead (MB) Value Bitmaps Update Bitmaps Fence Pointers Existence Bitmap n&=&m,&d&=&&distinct&domain&values UpBit UpBit-FP UCB In-place Compressed UpBit UpBit-FP UCB In-place Uncompressed update&bitvectors:&small&memory&footprint&when&compressed 28

29 UpBit vs.'scan Latency (ms) M M n&=&b,&d&=&&distinct&domain&values&(range) n&=&b,&d&vary&for&equality:&,,, M.5 B UpBit (range query) UpBit (equality query) Scan.% % % % we&have&the&classical&crossover&of&index&vs.&scan Selectivity shifts&when&the&query&needs&to&or&multiple&bitvectors&(range) 29

30 UpBit:'achieving'scalable'updates distribute&the&update&burden& update'bitvectors Thanks! efficient&bitvector&accesses fence'pointers avoid&redundant&bitwise&operations query2driven'merging'of'ub 3

31 Building'Access'Methods every&access&method &is&optimizing&for&the&tradeoff&between Tree Index Bitmap'Index Reads Updates Memory this&balance&forms&a&threeqway&tug&of&war 3