Basic
Java! / intro
to web search course / elasticsearch
Assignment:
indexing for web search
Your
task is to apply your IR skills to build a processing pipeline that
turns a Web site into structured knowledge. Your
system should take HTML pages as input, process them using the kind
of techniques that we have been looking at in the module, and output
an index of terms identified in the documents.
This
assignment comes in stages. Marks are given for each stage. You may
choose not to attempt some stages. You might also implement a system
that does not strictly follow the stages but will work in the same
way. The stages are as follows:
Input/Output
(10%) The
system must be able to read Web pages (a small number will do here
and they can be stored locally) and produce appropriately formatted
output. The Web pages should be processed one at a time using the
steps outlined below.
HTML
Parsing (10%) Before
the text can be analyzed it is necessary to get rid of the HTML tags.
The result will be plain text. Note that if you simply delete all
HTML tags, you will lose information such as meta tag keywords.
Therefore, I strongly suggest that you use some tool to perform this
task.
Pre-processing:
Sentence Splitting, Tokenization and Normalization (10%) The
next step should be to transform the input text into a normal form of
your choice.
Part-of-Speech
Tagging (10%) The
input should be tagged with a suitable part-of-speech tagger, so that
the result can then be processed in the next steps.
Selecting
Keywords (20%) One
aim of your system is to identify the words or phrases in the text
that are most useful for indexing purposes. Your system should remove
words which are not useful, such as very frequent words or stopwords.
You should develop a selection method, possibly using POS tags (e.g.
nouns and noun phrases) in combination with statistical/frequency
information (e.g. using term frequency).
Stemming
or Morphological Analysis (10%) Writing
word stems to the database rather than words allows to treat various
inflected forms of a word in the same way, i.e. bus and busses refer
to exactly the same thing even though they are di erent words.
Engineering
a Complete System (10%)
The nal system should have control over all the individual components
so that there is a single call and all the above steps will be
performed.
You
will have noticed that the percentages above only add up to 80%. This
is because one of the important aspects of the project is that your
work should be well documented and your code well commented. 20% of
your mark will come from this. You should submit:
A
description of your implementation: what the code does, and the
software you used
Unedited
and commented output from a run of the code submitted using these web
pages:
(feel
free to submit other runs as well, i.e. using Web pages of your own
choice)
A
short discussion of your solution focussing on functionality
implemented and possible improvements and extensions.
You
can implement your system either on the Linux
or the Windows machines. Perl, Java, Python, C/C++, and shell scripts
are good choices for this project, but
you are by no means restricted to those languages. Identify suitable
open-source tools that help you building your pipeline.
The post Assignment: indexing for web search appeared first on My Assignment Online.