User:OrenBochman/BetterSearch

1. (Also) index output HTML file?

1. solution:
   Either index these too, or run a filter to remove them. Some Strategies are:
   1. Discard all markup.
      1. A markup_filter/tokenizer could be used to discard markup.
      2. Lucene Tika project can do this.
      3. Other ready made solutions.
   2. Keep all markup
      1. Write a markup-analyzer that would be used to compress the page to reduce storage requirements.
         (interesting if one wants to also compress output for integrating into DB or Cache.
   3. Selective processing
      1. A table_template_map extension could be used in a strategy to identify structured information for deeper indexing.
      2. This is the most promising it can detect/filter out unapproved markup (Javascripts, CSS, Broken XHTML).

1. real-time "only" - slowly build index in background
2. offline "only" - used dedicated machine/cloud to dump and index offline.
3. dua - each time the lingustic component becomes significantly better (or there is a bug fix) it would be desire able to upgrade search. How this would be done would depend much on the architecture of the analyzer. One possible aproach would be
   1. production of a linguistic/entity data or a new software milestone.
   2. offline analysis from dump (xml,or html)
   3. online processing newest to oldest updates with a (Poisson wait time prediction model)

1. N-Gram analyzer is language independent.
2. A new Multilingual analyzer with a language detector can produced by
3. extract features from query and check against model prepared of line.
4. model would contain lexical feature such as:
   1. alphabet
   2. bi/trigram distribution.
   3. stop lists; collection of common word/pos/language sets (or lemma/language)
   4. normalized frequency statistics based on sampling full text from different languages..

1. in language with rich morphology this will reduce effectiveness of search. Hebrew, Arabic,
2. index Wiktionary so as to produce data for a "lemma analyzer".
   1. dumb lemma (bag with a representative)
   2. smart lemma (list ordered by frequency)
   3. quantum lemma (organized by morphological state and frequency)
3. lemma based indexing.
4. run a semantic disambiguation algorithm (tag )on disambiguate

• other benefits:

1. lemma based compression. (arithmetic coding based on smart lemma)
   1. indexing all lemmas
2. smart resolution of disambiguation page.
3. algorithm translate English to simple English.
4. excellent language detection for search.

• metrics:

1. extract amount of information contributed by a user
   1. since inception.
   2. in final version.

• media wiki manual
• extentions

• Search Extentions

highlights:

• Extension MWSearch
• Lucene Search
• Extension:EzMwLucene
• Extension:SphinxSearch

A quick review of the above is summarized as follows:

Mediawiki does not appear to have native search capabilities. It can be searched via external components (indexed and then searched) via three extensions:

1. Sphinx Search - for small sites (updated 2010)
2. Lucene Search - Lucene search for large sites
3. EzMwLucene - Easy Lucene search - an unadapted package from

MWSearch does not perform searches rather it provides integration with Lucene-search.

comitt capable developers, irc:#mediawiki

• Brion Vibber - lead dev
• Multichill
• Andrew Garrett - active paid developer
• Roan Kattouw - usability initiative, previously lead developer and maintainer of the MediaWiki API.
• Siebrand Mazeland

• Seb35
• Aryeh Gregor
• David Richfield
• Niklas Laxström - experienced MediaWiki developer

• zim offline format