# Java Notes: Example - WordFrequency

This program reads files of words and lists their frequencies. A file of words to ignore can also be supplied. This example illustrates the use of generic data structures (Sets, Maps, and ArrayList), regular expressions (as used by Scanner), and Comparators.

Exercises. A command-line interface is provided, but an interesting exercise is to write a GUI interface.

### A text command-line interface

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
    // File   : data-collections/wordfreq2/WordFrequencyCmd.java
    // Purpose: Main program to test WordCounter
    //          Prints word frequency in source file. Ignores words in ignore file.
    //          Uses Sets, Maps, ArrayList, regular expressions, BufferedReader.
    // Author : Fred Swartz - April 2007 - Placed in public domain.

    import java.io.*;
    import java.util.*;

    /////////////////////////////////////////////////////////////// WordFrequencyCmd
    class WordFrequencyCmd {
        //===================================================================== main
        public static void main(String[] unused) {
            Scanner in = new Scanner(System.in);

            try {
                //... Read two file names from the input.
                System.out.println("Name of file containing text to analyze:");
                File inputFile = new File(in.nextLine());

                System.out.println("Name of file containing words to ignore:");
                File ignoreFile = new File(in.nextLine());

                //... Supply two files to WordCounter.
                WordCounter counter = new WordCounter();
                counter.ignore(ignoreFile);
                counter.countWords(inputFile);

                //... Get the results.
                String[] wrds   = counter.getWords(WordCounter.SortOrder.BY_FREQUENCY);
                int[] frequency = counter.getFrequencies(WordCounter.SortOrder.BY_FREQUENCY);

                //... Display the results.
                int n = counter.getEntryCount();
                for (int i=0; i<n; i++) {
                    System.out.println(frequency[i] + " " + wrds[i]);
                }

                System.out.println("\nNumber of input words: " + counter.getWordCount());
                System.out.println("Number of unique words: " + n);

            } catch (IOException iox) {
                System.out.println(iox);
            }
        }
    }

### The "Model" -- Word frequency without any user interface

The model (the logic of the program without any user interface) is implemented primarily in the WordCounter class, which uses utility classes: ComparatorFrequency and ComparatorAlphabetic to sort, and Int to record the frequency counts.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    185
    186
    187
    188
    189
    190
    191
    192
    193
    194
    195
    196
    197
    198
    199
    200
    201
    202
    203
    204
    // File   : data-collections/wordfreq2/WordCounter.java
    // Purpose: Provides methods to read ingore file, input file, get results.
    //          Computes the frequency for each word.
    // Author : Fred Swartz - April 2007 - Placed in public domain.

    import java.io.*;
    import java.util.*;
    import java.util.regex.*;

    /** Computes word frequency in source file; ignores words in ignore file.
     * Uses generic Sets, Maps, ArrayLists, regular expressions, Scanner.
     * @author Fred Swartz
     * @version 2007-05-06
     */
    public class WordCounter {
        //================================================================ constants
        private static final Comparator<Map.Entry<String, Int>> SORT_BY_FREQUENCY =
                new ComparatorFrequency();
        private static final Comparator<Map.Entry<String, Int>> SORT_ALPHABETICALLY =
                new ComparatorAlphabetic();
        public enum SortOrder {ALPHABETICALLY, BY_FREQUENCY}

        //=================================================================== fields
        Set<String>      _ignoreWords;   // Words to ignore.
        Map<String, Int> _wordFrequency; // Words -> frequency
        int              _totalWords;    // Total source words.

        //============================================================== constructor
        /** Constructor */
        public WordCounter() {
            _ignoreWords   = new HashSet<String>();
            _wordFrequency = new HashMap<String, Int>();
            _totalWords    = 0;
        }

        //=================================================================== ignore
        /**
         * Reads file of words to ignore. Ignore words are added to a Set.
         *  The IOException is passed to caller because we certinaly don't
         *  know what the user interface issue is.
         *
         * @param ignoreFile File of words to ignore.
         */
        public void ignore(File ignoreFile) throws IOException {
            Scanner ignoreScanner = new Scanner(ignoreFile);
            ignoreScanner.useDelimiter("[^A-Za-z]+");

            while (ignoreScanner.hasNext()) {
                _ignoreWords.add(ignoreScanner.next());
            }
            ignoreScanner.close();  // Close underlying file.
        }

        //=================================================================== ignore
        /**
         * Takes String of words to ignore. Ignore words are added to a Set.
         *
         * @param ignore String of words to ignore.
         */
        public void ignore(String ignoreStr) {
            Scanner ignoreScanner = new Scanner(ignoreStr);
            ignoreScanner.useDelimiter("[^A-Za-z]+");

            while (ignoreScanner.hasNext()) {
                _ignoreWords.add(ignoreScanner.next());
            }
        }

        //=============================================================== countWords
        /** Record the frequency of words in the source file.
         *  May be called more than once.
         *  IOException is passed to caller, who might know what to do with it.
         *@param File of words to process.
         */
        public void countWords(File sourceFile) throws IOException {
            Scanner wordScanner = new Scanner(sourceFile);
            wordScanner.useDelimiter("[^A-Za-z]+");

            while (wordScanner.hasNext()) {
                String word = wordScanner.next();
                _totalWords++;

                //... Add word if not in map, otherewise increment count.
                if (!_ignoreWords.contains(word)) {
                    Int count = _wordFrequency.get(word);
                    if (count == null) {    // Create new entry with count of 1.
                        _wordFrequency.put(word, new Int(1));
                    } else {                // Increment existing count by 1.
                        count.value++;
                    }
                }
            }
            wordScanner.close();  // Close underlying file.
        }

        //=============================================================== countWords
        /** Record the frequency of words in a String.
         *  May be called more than once.
         *@param String of words to process.
         */
        public void countWords(String source) {
            Scanner wordScanner = new Scanner(source);
            wordScanner.useDelimiter("[^A-Za-z]+");

            while (wordScanner.hasNext()) {
                String word = wordScanner.next();
                _totalWords++;

                //... Add word if not in map, otherewise increment count.
                if (!_ignoreWords.contains(word)) {
                    Int count = _wordFrequency.get(word);
                    if (count == null) {    // Create new entry with count of 1.
                        _wordFrequency.put(word, new Int(1));
                    } else {                // Increment existing count by 1.
                        count.value++;
                    }
                }
            }
        }

        //============================================================= getWordCount
        /** Returns number of words in all source file(s).
         *@return Total number of words proccessed in all source files.
         */
        public int getWordCount() {
            return _totalWords;
        }

        //============================================================ getEntryCount
        /** Returns the number of unique, non-ignored words, in the source file(s).
         *  This number should be used to for the size of the arrays that are
         *  passed to getWordFrequency.
         *@return Number of unique non-ignored source words.
         */
        public int getEntryCount() {
            return _wordFrequency.size();
        }

        //========================================================= getWordFrequency
        /** Stores words and their corresponding frequencies in parallel array lists
         *  parameters.  The frequencies are sorted from low to high.
         * @param words Unique words that were found in the source file(s).
         * @param counts Frequency of words at corresponding index in words array.
         */
        public void getWordFrequency(ArrayList<String> out_words,
                ArrayList<Integer> out_counts) {
            //... Put in ArrayList so sort entries by frequency
            ArrayList<Map.Entry<String, Int>> entries =
                    new ArrayList<Map.Entry<String, Int>>(_wordFrequency.entrySet());
            Collections.sort(entries, new ComparatorFrequency());

            //... Add word and frequency to parallel output ArrayLists.
            for (Map.Entry<String, Int> ent : entries) {
                out_words.add(ent.getKey());
                out_counts.add(ent.getValue().value);
            }
        }

        //================================================================= getWords
        /** Return array of unique words, in the order specified.
         * @return An array of the words in the currently selected order.
         */
        public String[] getWords(SortOrder sortBy) {
            String[] result = new String[_wordFrequency.size()];
            ArrayList<Map.Entry<String, Int>> entries =
                    new ArrayList<Map.Entry<String, Int>>(_wordFrequency.entrySet());
            if (sortBy == SortOrder.ALPHABETICALLY) {
                Collections.sort(entries, SORT_ALPHABETICALLY);
            } else {
                Collections.sort(entries, SORT_BY_FREQUENCY);
            }

            //... Add words to the String array.
            int i = 0;
            for (Map.Entry<String, Int> ent : entries) {
                result[i++] = ent.getKey();
            }
            return result;
        }

        //=========================================================== getFrequencies
        /** Return array of frequencies, in the order specified.
         * @return An array of the frequencies in the specified order.
         */
        public int[] getFrequencies(SortOrder sortBy) {
            int[] result = new int[_wordFrequency.size()];
            ArrayList<Map.Entry<String, Int>> entries =
                    new ArrayList<Map.Entry<String, Int>>(_wordFrequency.entrySet());
            if (sortBy == SortOrder.ALPHABETICALLY) {
                Collections.sort(entries, SORT_ALPHABETICALLY);
            } else {
                Collections.sort(entries, SORT_BY_FREQUENCY);
            }

            //... Add words to the String array.
            int i = 0;
            for (Map.Entry<String, Int> ent : entries) {
                result[i++] = ent.getValue().value;
            }
            return result;
        }
    }

## Comparators

The key-value (`Map.Entry`) pairs in a `HashMap` are not sorted, but users would certainly like to see them sorted, either alphabetically by word, or by frequency.

Because sorts are usually based on comparison of two values at a time, all that is needed is a way to compare two values. That's what a `Comparator` does -- it defines a `compare()` method that tells how two values compare.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
    // File   : data-collections/wordfreq2/ComparatorAlphabetic.java
    // Purpose: A comparator to sort Map.Entries alphabetically.
    // Author : Fred Swartz - March 2005 - Placed in public domain.

    import java.util.*;

    /////////////////////////////////////////////////////// class ComparatorAlphabetic
    /** Order words alphabetically. */
    class ComparatorAlphabetic implements Comparator<Map.Entry<String, Int>> {
        public int compare(Map.Entry<String, Int> entry1
                         , Map.Entry<String, Int> entry2) {
            return entry1.getKey().compareTo(entry2.getKey());
        }
    }

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
    // File   : data-collections/wordfreq2/ComparatorFrequency.java
    // Purpose: A comparator to sort Map.Entries by frequency.
    // Author : Fred Swartz - April 2007 - Placed in public domain.

    import java.util.*;

    ////////////////////////////////////////////////////// class ComparatorFrequency
    /** Order words from least to most frequent, put ties in alphabetical order. */
    class ComparatorFrequency implements Comparator<Map.Entry<String, Int>> {
        public int compare(Map.Entry<String, Int> obj1
                         , Map.Entry<String, Int> obj2) {
            int result;
            int count1 = obj1.getValue().value;
            int count2 = obj2.getValue().value;
            if (count1 < count2) {
                result = -1;

            } else if (count1 > count2) {
                result = 1;

            } else {
                //... If counts are equal, compare keys alphabetically.
                result = obj1.getKey().compareTo(obj2.getKey());
            }
            return result;
        }
    }

## Mutable Integers

Because only objects (not primitives) can be stored in Collections data structures, the integer count must be an object, not just an `int`. The natural choice would be the `Integer` wrapper type, but it is immutable, which means that once an `Integer` object is created, its value can never be changed. But here we need to update the value, so here's a class that stores an int that can be changed.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
    // File   : data-collections/wordfreq2/Int.java
    // Purpose: Simple value class to hold a mutable int.
    // Author : Fred Swartz - March 2005 - Placed in public domain.

    //////////////////////////////////////////////////////////////// value class Int
    /** Utility class to keep int as Object but allow changes (unlike Integer).
     * Java collections hold only Objects, not primitives, but need to update value.
     * The intention is that the public field should be used directly.
     * For a simple value class this is appropriate.
     */
    class Int {
        //=================================================================== fields
        public int value;  // Just a simple PUBLIC int.

        //============================================================== constructor
        /** Constructor
            @param value Initial value. */
        public Int(int value) {
            this.value = value;
        }
    }