COMP 776: Computer Vision

Transcription

1 COMP 776: Computer Vision

2 Basic Info Instructor: Svetlana Lazebnik Office hours: By appointment, SN 219 Textbook: Forsyth & Ponce, Computer Vision: A Modern Approach Class webpage:

3 Today Introduction to computer vision Course overview Course requirements

4 Why study computer vision? Images and video are everywhere! Personal photo albums Movies, news, sports Surveillance and security Medical and scientific images

5 Vision is useful Vision is interesting Vision is difficult Why study computer vision? Half of primate cerebral cortex is devoted to visual processing Achieving human-level visual perception is probably AI-complete

6 Origins of computer vision L. G. Roberts, Machine Perception of Three Dimensional Solids, Ph.D. thesis, MIT Department of Electrical Engineering, 1963.

7 Connections to other disciplines Artificial Intelligence Robotics Machine Learning Computer Vision Computer Graphics Psychology Neuroscience Image Processing

8 Applications of computer vision Factory inspection Reading license plates, checks, ZIP codes Monitoring for safety (Poseidon) Surveillance Sources: K. Grauman, L. Fei-Fei Autonomous driving, robot navigation Driver assistance (collision warning, lane departure warning, rear object detection)

9 Applications of computer vision Assistive technologies Entertainment (Sony EyeToy) Movie special effects Digital cameras (face detection for setting focus, exposure) Visual search (MSR Lincoln) Sources: K. Grauman, L. Fei-Fei

10 Applications of computer vision For more information on the computer vision industry:

11 The goal of computer vision To perceive the world behind the picture

12 The goal of computer vision To perceive the world behind the picture

13 The goal of computer vision To perceive the world behind the picture What exactly does this mean? Vision as a source of metric 3D information Vision as a source of semantic information

14 Real-time stereo Vision as measurement device Structure from motion Multi-view stereo for community photo collections NASA Mars Rover Pollefeys et al. Goesele et al. Amazing success story!

15 but why do Learning for Vision? What if I don t care about this wishy-washy recognition stuff? I just want to make my robot go! Small Reason: For measurement, other sensors are often better (in DARPA Grand Challenge, vision was barely used!) For navigation, you still need to learn! Big Reason: The goals of computer vision (what + where) are in terms of what humans care about. Slide credit: A. Efros

16 Vision as a source of semantic information slide credit: Fei-Fei, Fergus & Torralba

17 Object categorization sky building flag banner bus face street lamp bus wall cars slide credit: Fei-Fei, Fergus & Torralba

18 Scene and context categorization outdoor city traffic slide credit: Fei-Fei, Fergus & Torralba

19 Qualitative spatial information slanted non-rigid moving object vertical rigid moving object horizontal rigid moving object slide credit: Fei-Fei, Fergus & Torralba

20 Challenges: viewpoint variation Michelangelo slide credit: Fei-Fei, Fergus & Torralba

21 Challenges: illumination image credit: J. Koenderink

22 Challenges: scale slide credit: Fei-Fei, Fergus & Torralba

23 Challenges: deformation Xu, Beihong 1943 slide credit: Fei-Fei, Fergus & Torralba

24 Challenges: occlusion Magritte, 1957 slide credit: Fei-Fei, Fergus & Torralba

25 Challenges: background clutter

26 Challenges: object intra-class variation slide credit: Fei-Fei, Fergus & Torralba

27 Challenges: local ambiguity slide credit: Fei-Fei, Fergus & Torralba

28 Challenges or opportunities? Images are confusing, but they also reveal the structure of the world through numerous cues Our job is to interpret the cues! Image source: J. Koenderink

29 Depth cues: Linear perspective

30 Depth cues: Aerial perspective

31 Depth ordering cues: Occlusion Source: J. Koenderink

32 Shape cues: Texture gradient

33 Shape and lighting cues: Shading Source: J. Koenderink

34 Position and lighting cues: Cast shadows Source: J. Koenderink

35 Grouping cues: Similarity (color, texture, proximity)

36 Grouping cues: Common fate Image credit: Arthus-Bertrand (via F. Durand)

37 Bottom line Perception is an inherently ambiguous problem Many different 3D scenes could have given rise to a particular 2D picture Image source: F. Durand

38 Bottom line Perception is an inherently ambiguous problem Many different 3D scenes could have given rise to a particular 2D picture Possible solutions Bring in more constraints (more images) Use prior knowledge about the structure of the world Need both exact measurements and statistical inference! Image source: F. Durand

39 Course overview I. Early vision: Image formation and processing II. Mid-level vision: Grouping and fitting III. Multi-view geometry IV. Recognition V. Advanced topics

40 I. Early vision Basic image formation and processing * = Cameras and sensors Light and color Linear filtering Edge detection Feature extraction: corner and blob detection

41 Fitting and grouping II. Mid-level vision Alignment Fitting: Least squares Hough transform RANSAC

42 III. Multi-view geometry Stereo Epipolar geometry Tomasi & Kanade (1993) Affine structure from motion Projective structure from motion: Here be dragons!

43 IV. Recognition Patch description and matching Clustering and visual vocabularies Bag-of-features models Classification Sources: D. Lowe, L. Fei-Fei

44 V. Advanced Topics Time permitting Segmentation Face detection Articulated models Motion and tracking

45 Course requirements Philosophy: computer vision is best experienced hands-on Programming assignments: 40% Three or four assignments Expect the first one in a couple of weeks Brush up on your MATLAB skills (see web page for tutorial) Final project: 50% Putting several pieces together List of options will be posted in the next few weeks (some great ideas already from Gary Bishop) Expect to commit to a project idea by the end of February Participation: 10% Ask questions Answer questions Give me feedback: I m learning too!

46 Collaboration policy Feel free to discuss assignments with each other, but coding must be done individually Feel free to incorporate code or tips you find on the Web, provided this doesn t make the assignment trivial and you explicitly acknowledge your sources Remember: I can Google too!

47 Next week Homework: MATLAB tutorial (self-study, not collected) Reading: cameras and image formation (F&P chapter 1)