Web Information Retrieval Dipl.-Inf. Christoph Carl Kling

Transcription

1 Institute for Web Science & Technologies University of Koblenz-Landau, Germany Web Information Retrieval Dipl.-Inf. Christoph Carl Kling

2 Exercises WebIR ask questions! 2 of 49

3 Clustering Coefficient (!= Transitivity ratio) Definitions: global clustering coefficient: local clustering\frac{ coefficient: 3 of 49

4 Clustering Coefficient (!= Transitivity ratio) 4 of 49

5 Clustering Coefficient (!= Transitivity ratio) 5 of 49

6 IR Document collection d1: Marcus try assassin Caesar. d2: Marcus Rome. d3: Caesar rule. Rome loyal Caesar hate. d4: loyal. People try assassin rule loyal. Terms: marcus,try,assassin,caesar,rome,rule,loyal,hate,people TF-IDF vectors: d1: (1,1,1,1,0,0,0,0,0) d2: (1,0,0,0,1,0,0,0,0) d3: (0,0,0,1,.5,.5,.5,1,0) d4: (0,.5,.5,0,0,.5,1,0,1) 6 of 49

7 Exercise: (Re)calculate the tf-idf scores for d1-d5. We use the log to base 2 (actually any log works). 7 of 49

8 Exercise: Prove that cosine-similarity and Euklidean distance yield an identical relative distance measure (e.g. between documents) when all document and query vectors are normalised to be of length 1. 8 of 49

9 Exercise: Prove that cosine-similarity and Euklidean distance yield an identical relative distance measure (e.g. between documents) when all document and query vectors are normalised to be of length 1. d7 d6 d6 normalise caesar d8 caesar assassin d7 d8 assassin 9 of 49

10 d7 d6 d6 normalise caesar d8 caesar assassin d7 d8 assassin 10 of 49

11 y * euklid cos x * * for normalised vectors of length 1 11 of 49

12 Given the search request Assassinate people The query is a binary vector again: q: (0,0,1,0,0,0,0,0,1) 12 of 49

13 q: (0,0,1,0,0,0,0,0,1) This time, we don't know which docs are relevant. Instead, we look at which documents have both a high score for the terms assassin and people. The relevance now is just the sum of the tf-idf weights for every word in the query. 13 of 49

14 The relevance now is just the sum of the tf-idf weights for every word in the query. For every document, we have to sum over its tf-idf scores for the words in the query. The tf-idf weights are stored in inverted indexes. 14 of 49

15 The tf-idf weights are stored in inverted indexes. assassin doc13: doc4: doc99: doc767: people doc99: doc6: doc13: doc4: caesar doc767: doc41: doc52: doc4: Score: sum of tfidf weights for search terms. For the query assassin people : doc13 has score 1.28 doc4 has score 0.33 doc99 has score of

16 The tf-idf weights are stored in inverted indexes. Score: sum of tfidf weights for search terms. This means, we have to do a join in a database, where we look up the score of every every document for all query terms and calculate the sum. Sounds bad. 16 of 49

17 To speed up the merging of tf-idf weights, documents can be ordered and we jump to buckets of document IDs when searching for an ID to join. (This is just what you did when searching a name in a phone book.) assassinate: people: of 49

18 18 of 49

19 The tf-idf weights are stored in inverted indexes. Our query is now: people assassin caesar By ordering the documents by their tfidf-score, we can speed up the process of calculating the K documents with the highest sum of tf-idf scores: assassin people g: of 49 caesar

20 20 of 49

21 assassin people g: of 49 caesar

22 assassin people g: of 49 caesar

23 assassin people g: of 49 caesar

24 assassin people g: of 49 caesar

25 25 of 49

26 Non-random-access algorithm: We only look at the first i rows. assassin people g: of 49 caesar

27 assassin 0.5 people 0.55 caesar Top 2 documents: Threshol Candidates: 27 of 49

28 assassin 0.5 people 0.55 caesar Top 2 documents: Threshol 1.4 Candidates: 28 of 49

29 assassin Top 2 documents: people 0.55 Threshol 1.4 Candidates: caesar 29 of 49

30 assassin Top 2 documents: people 0.55 Threshol 0.95 Candidates: 0.5 caesar 30 of 49

31 assassin people 0.55 caesar Top 2 documents: Threshol 0.95 Candidates: of 49

32 assassin people 0.55 caesar Top 2 documents: Threshol 0.75 Candidates: of 49

33 assassin people g: 0.55 caesar Top 2 documents: Threshol 0.75 Candidates: of 49

34 assassin people g: 0.55 caesar Top 2 documents: Threshol 0.6 Candidates: 0.7 g: 34 of 49

35 assassin people g: 0.55 caesar Top 2 documents: Candidates: 0.7 g: Threshol (min score in top-k) is bigger than the threshold of 0.6! We found the top-k documents! 35 of 49

36 Markov Chain Markov chain: We are interested in ergodic Markov chains: - homogeneous (transition probabilities fixed) - irreducible (every state always reachable) - aperiodic (greatest common divisor = 1 for recurrence) 36 of 49

37 Page Rank Transition probability: 1/number of links 37 of 49

38 Page Rank Exercise: Is this Markov chain ergodic? 38 of 49

39 Page Rank Random jump with ( normal jump with (1-e) = 0.7 ) the markov chain becomes ergodic! 39 of 49

40 Page Rank Transition matrix P (see board) Initial state 40 of 49

41 Page Rank Exercise: Calculate Will a converge? And why? 41 of 49

42 Markov Chain Authority scores Hub scores 42 of 49

43 Hubs and Authorities (or HITS) algorithm Authority scores Hub scores 43 of 49

44 Hubs and Authorities (or HITS) algorithm Authority scores Hub scores These vectors have to be normalised to sum to 1! 44 of 49

45 Hubs and Authorities (or HITS) algorithm Authority scores Hub scores :) 45 of 49

46 Hubs and Authorities (or HITS) algorithm Exercise: Calculate and normalise x and y after each step so that each sums to 1 (that is, the values can be interpreted vector as probabilities). 46 of 49

47 Page Rank Exercise: Hubs and Authorities - unlike Page Rank - does not take the whole web graph, but only a subset. Describe how the Hubs and Authority algorithm works! 47 of 49

48 48 of 49

49 49 of 49