Yoshikoder projects can have an associated content analysis dictionary. A dictionary is a tree structure of categories each of which may contain patterns. Patterns are strings that can match single words in a document. Categories simply organize them in a hierarchical way.
+You can add, edit and remove categories and patterns using Dictionary>Add Category and Dictionary>Add Pattern Dictionary>Edit Entry and Dictionary>Remove Entry, or by using the toolbar icons. You can also move categories and patterns around the dictionary by drag and drop.
+If you have a lot of patterns to add, use the Dictionary>Add Patterns function to add them all at once. You'll get a text window into which you can paste as many words as you like. Whatever you paste will be split up and added as new patterns within the category you have selected.
+Patterns may contain the wildcard character * which means 'zero or more other letters'. Hence shout* matches 'shout', 'shouting', and 'shouted'. The star can appear in any part of the string or more than once.
Matching is not case sensitive, so 'Shout' and 'shout are both matched by the pattern shout.
Although patterns cannot be duplicated within a category, wildcarding may nevertheless allow double counting. You can check for this by using the highlighting and concordance functions.
+Dictionaries can be saved and loaded independently of projects using Dictionary>Open and Dictionary>Save As. When you open a new dictionary it replaces the old one, so don't forget to save a copy if you want to use it again.
+Dictionary>Open supports two dictionary formats: the native Yoshikoder format and VBPro.
+The native Yoshikoder dictionary format appears in files ending in '.ykd'. Web browsers sometimes also append '.xml' when they download such files but you can just remove that from the file's name.
+VBPro format is a a very simple flat dictionary specification which surrounds category names with '>>' and '<<' and lists patterns one per line. For example:
+>>Fruit<<
+ pear
+ tomato
+ >>Vegetables<<
+ carrot
+ potatorepresents a two category dictionary with four patterns. Hierarchical structure cannot be represented in this format.
+VBPro format was designed so that it is easy to construct dictionaries manually in a text editor.
+ + + diff --git a/client/onlinehelp/Documents.html b/client/onlinehelp/Documents.html new file mode 100644 index 00000000..35541a23 --- /dev/null +++ b/client/onlinehelp/Documents.html @@ -0,0 +1,41 @@ + + + + + + +Yoshikoder deals with plain text documents. If you have documents in other formats, e.g. Word, or PDF you should convert them first. (See 'Converting to plain text' below).
+When you add documents to a project, e.g. using Documents>Add Documents their names appear in the documents list. Note that when you add a document to a project you are only adding a reference to it. This has two implications:
+First, if you close the application and then move or delete the original document, Yoshikoder will not be able to find it again when it re-opens. If that happens you should remove the dangling reference from the document list and re-add the document from its new location.
+Second, when you remove a document from the Yoshikoder you do not affect the original file. It is still there in your filesystem.
+To remove documents from a project, selectthem and use Documents>Remove Documents
+You can also export a copy of any document in one of two Unicode formats using the Documents>Export menu item.
+Documents have a location in the file system; a title, by which they are known within a project; an encoding, which records what character set the document file uses; and a locale, which indicates what language the document is in. These are set to default values specified in the Preferences when documents are added, but can be altered later using Documents>Edit Document.
+Note that altering the file encoding is not recommended unless the current one has generated nonsense in the text window. Altering the locale will have little if any effect on tokenisation in Western languages.
+Most applications will allow you to 'save as' text. If you have a lot of documents it may be easier to use a tool to convert them in one go. The YKConverter might be useful for this.
+If, after conversion to plain text you find strange looking characters in your documents when you add them this is a sign that Yoshikoder's expected encoding (sometimes called 'character set') is not same as the document's encoding. If this happens you should determine the documents' encoding, set the Yoshikoder's default encoding appropriately in the Preferences, and re-add the documents.
+Alternatively you can convert the documents into the Yoshikoder's default encoding before adding them (provided you are confident that it will be able to represent all the characters in the documents) The Reencoder might be helpful for this.
+ + + diff --git a/client/onlinehelp/FAQ.html b/client/onlinehelp/FAQ.html new file mode 100644 index 00000000..0b2b3b05 --- /dev/null +++ b/client/onlinehelp/FAQ.html @@ -0,0 +1,38 @@ + + + + + + +Probably because when you ran the report you had just one category selected in the interface. The application used not to care what you had selected, but now it does. To get the old behavior back, select the dictionary root before you run reports.
+Probably the orginal document has moved from where it was when you added it to the project. Remove the phantom document references and re-add them to the project.
+Not yet. Sorry.
+The easiest way to figure out what encoding a document is in is to open it in a browser and play with the encoding menu (usually found under 'View' or a similarly named menu) until it looks right. Mac and Unix users can also use the 'file' command at the shell prompt. Everybody can preview and convert documents between encodings en masse using Reencoder.
+They went away. It was always hard to provide sensible semantics for them - should they be averaged within category or summed, or weighted by document proportions? Scores are now something you do to a dictionary report after it's made.
+ + + diff --git a/client/onlinehelp/HighlightingandConcordances.html b/client/onlinehelp/HighlightingandConcordances.html new file mode 100644 index 00000000..4b5e3b20 --- /dev/null +++ b/client/onlinehelp/HighlightingandConcordances.html @@ -0,0 +1,32 @@ + + + + + + +If you want to see all the matches for a pattern or a category in the dictionary, select the dictionary entry and use Highlight>Highlight Entry and Highlight>Remove Highlights to remove all highlighting. You can also change the color of the highlighting from the same menu.
+Concordances are a way to see the contexts of your matches in one place. If you select a dictionary entry and select one or more documents then Concordance>Make Concordance will create a keyword in context view of all the entries' matches in each documents in the concordance window. If you want more words either side of the dictionary matches, change the width of the concordance in the Preferences.
+You can save concordances using Concordance>Save Concordance, reopen them with Concordance>Open Concordance or export them in other formats.
+It is also possible to run dictionary analyses over current of constructed concordances. See the Reports help pages for details.
+ + + diff --git a/client/onlinehelp/Projects.html b/client/onlinehelp/Projects.html new file mode 100644 index 00000000..aba09a89 --- /dev/null +++ b/client/onlinehelp/Projects.html @@ -0,0 +1,32 @@ + + + + + + +A project is a combination of a dictionary and a set of documents. When you open the application, the last project you worked on will be displayed, if you saved it. Otherwise you start with an empty dictionary and no documents.
+Projects are contain the complete dictionary but only file system-specific references to documents. As a result you cannot zip up your project file, send it to a colleague and expect it to work for them. You can, however save the dictionary using Dictionary>Save As which your colleague can open in their own project.
+It is also not advisable to move the project file (usually called suffixed '.ykp') around. You can, however, use File>New Project to create new project elsewhere. You will be prompted whether you want to keep references to the current project's documents.
+ + + diff --git a/client/onlinehelp/Reports.html b/client/onlinehelp/Reports.html new file mode 100644 index 00000000..308f8caf --- /dev/null +++ b/client/onlinehelp/Reports.html @@ -0,0 +1,46 @@ + + + + + + +The application offers two kinds of reports, those that appear in the interface, and those that are computed and go straight to a file for further analysis.
+Perhaps the simplest report shows the frequencies of each word type in the current document in a dialog. To create this report use is Reports>Count Words>Current Document.
+To get the same information for multiple documents, select the documents you are interested in and use Reports>Count Words>Multiple Documents. You will be prompted for a file, which will for resource reasons be limited to UTF-8 encoded CSV (a widely readable spreadsheet format). The resulting file has document names as rows and word types as columns.
+Be aware that these files can get big and can be difficult to deal with. If you are serious about counting lots of words, a more efficient method and representation is recommended, e.g. the one used by JFreq.
+To apply the dictionary to the current document only use Reports>Apply Dictionary>Current Document. The results are shown in a dialog box.
+If you are only interested in applying one part of the dictionary, select a category entry of interest before you apply the dictionary to the current document. Then only matches to that category and the ones below it will be computed. If the top level category is selected, a complete analysis will be computed.
+If you want the same information for a larger number of documents, select the documents and the appropriate dictionary category and choose a file for the results. For this report, it is possible to choose either UTF-8 encoded CSV or MS Excel as a format. (If you have hundreds of categories, stick with the former.)
+It is possible to do statistical comparisons of two documents using 'relative risk', a quantity used in epidemiology. In this report the dictionary, or a selected subset of its entries, is applied to two documents using Reports>Apply Dictionary>Compare Document Pair. The report computes the relative probability of seeing each category in each document, controlling for their document lengths.
+This is best seen with an example. Assume a dictionary containing the category 'Gun control' and two documents 'Lib' containing 1200 words, of which 32 match patterns in 'Gun control' and 'Cons' containing 900 words, of which 10 match patterns in 'Gun control'. The estimated probability of seeing a 'Gun control' match in 'Lib' is 32/1200 which is about 0.027. For 'Cons' it is about 0.012. The ratio is about 2.4 so 'Lib' uses this category about 2.4-1 x 100 = 140% more often than 'Cons'. Or just more than twice as much, if you prefer.
+The report also computes a confidence interval around this ratio, marking 95% confidence intervals that do not include 1, meaning that 'Gun control' is used at an equal rate in both documents, with a single asterisk. (In the example above, the confidence interval would exclude 1).
+It is also possible to treat concordances as documents and apply the entire dictionary to them. This can be used to characterise the content of local contexts of matches. To apply the dictionary to a concordance created with Concordance>Make Concordance, use Reports>Apply Dictionary>Current Concordance. The left and right hand sides of the concordance are treated like a single document and the report appears in a dialog.
+If you want to do this analysis for a larger number of documents, then select the documents you are interested in, and the dictionary entry you want to use to define the concordance, and then use Reports>Apply Dictionary>Multiple Concordances. You will be prompted for a file to hold the results, which may be either UTF-8 encoded CSV or MS Excel. In the results, each row is labeled by the document whose text was used to generate the concordance, and each column is the name of a category in the dictionary. All categories are used.
+The duplicate pattern report generated by Reports>Duplicate Report scans the dictionary and reports all patterns that occur in more than one category along with the categories they appear in. This can be useful to ensure minimal levels of double counting in reports.
+ + + diff --git a/client/onlinehelp/Tokenization.html b/client/onlinehelp/Tokenization.html new file mode 100644 index 00000000..0fe4442d --- /dev/null +++ b/client/onlinehelp/Tokenization.html @@ -0,0 +1,51 @@ + + + + + + +You can add your own tokenizers for particular languages using the Tokenizers tab in the Preferences. If you want to build your own tokenizer, read on.
+Tokenizers are java classes that implement the interface 'Tokenizer'
+ public interface Tokenizer {
+ public TokenList getTokens(String txt) throws TokenizationException;
+ public Locale[] getLocales();
+ }You can compile against the Yoshikoder codebase from sourceforge. Here 'TokenList' trivially extends 'java.util.List'. The 'Token' interface is
+ public interface Token {
+ public String getText();
+ public int getStart();
+ public int getEnd();
+ }A 'Token' has some text, the token itself, and a span of characters. In a given document if a token spans the third through fifth character, then 'getStart' should return 2 (counting from 0), and 'getEnd' should return 5.
+The jar file is available that provides these interfaces and some default implementations: 'TokenImpl', 'TokenListImpl', and 'TokenizationException'. You should link to this jar when you compile your code and include it in deployments.
+Tokenizers should be deployed as Jar files. There must be a properties file at the top level of the Jar called 'tokenizer.properties'. The contents must include 'name' - the short name of the tokenizer, 'classname' - the fully qualified name of the 'Tokenizer' implementation, and 'locales' - a space separated list of java locales, specified in the same manner as the 'toString' method on 'java.util.Locale'. A 'description' is optional. A sample 'tokenizer.properties' file is shown below:
+ name=Simp. Chinese
+ description=Simplified Chinese Tokenizer
+ # Specify [lang] as well as [lang]_[country] locales where necessary
+ locales=zh_CN zh
+ classname=com.mandarintools.SimplifiedChineseTokenizerWelcome to the Yoshikoder help pages.
+