Two
electrical engineering professors from the Jacobs School of Engineering, Gert Lanckriet and Nuno Vasconcelos,
have
won spots in the 2013 Yahoo! Faculty Research and Engagement Program (FREP).
This
program funds help academics across the globe collaborate with Yahoo! research
scientists on new, exciting internet research studies and experiments.
A
summary of Lanckriet’s project is below.
Machine Learning with Highly Unstructured
Multimedia DataAn increasingly
important aspect of information in today’s world is that it is highly unstructured
and heterogeneous. For example, a search for “miss world 2013” yields
fundamentally different modes of information about the topic via news articles,
blogs, videos, pictures etc. While there are some commonalities in the
information provided in these modalities (i.e., they are about the beauty
pageant), the nature of information differs widely amongst the modalities.
Videos and pictures
are innately visual in nature, and differ from news articles and blogs which
are textual in nature. Even within the textual sources, there exist
differences: news articles are more factual in nature, while blogs are
opinionated. In multimedia applications, such information is often available in
a variety of unstructured formats: blogs may contain one or more pictures, or
none, embedded videos, audio tracks, and/or tweets, or not, etc.
Ideally, an algorithm
to make inferences from the data (e.g., whether or not the judges were fair in
deciding the winner) would be given access to all available information
sources, since each source may bring some unique information to the table.
Machine learning with such unstructured, heterogeneous data is challenging and
an active topic of research. Existing approaches assume a rigidly structured
input format. They cannot accommodate missing modalities (incomplete
information) or modalities with multiple instances (over-complete information).
In this project, we intend to design and develop principled approaches to
learning from heterogeneous, unstructured multimedia data.
One of the problems that we are interested in is artist recommendation. Current
approaches rely on either collaborative filtering methods (relying on the
wisdom of the crowd to determine if two artists are similar), or content based
methods (two artists are considered similar if the audio content of their songs
are similar) to determine if two artists are similar. We are interested
in developing a multimodal artist similarity, based on heterogeneous
unstructured social and multimedia data that relates to artists. This can
include band videos and pictures (which implicitly provide information about
the “look-and-feel” of an artist), tweets, reviews, and blogs about the
artists’ music (providing opinions and information about artists), lyrics,
their social network of listeners, etc. We believe that combining all this
information through a more accurate multimodal similarity measure will improve
recommendation compared to using content or collaborative filtering only.
.jpg)
.jpg)
