Java: Tokenizing Delimiters

Purpose

When tokenizing a string, the delimiters are ignored by default. However, it may be useful to get them returned as tokens too. Each delimiter will be returned as a one-character string.

Constructor

Specify that you want the delimiters returned by specifying true in the third parameter of the constructor.

StringTokenizer st = new StringTokenizer(s, d, true);

Creates a StringTokenizer for the String s using delimiters from the String d

Example: Tokenizing an expression

Here is an example method that tokenizes a string, making all delimiters except whitespace into tokens.

 private String tokenizeEx(String expression, String delims) {
    StringBuffer sb = new StringBuffer(100);
    StringTokenizer scanner;    
    scanner = new StringTokenizer(expression, delims, true);
    
    while (scanner.hasMoreTokens()) {
       String aWord = scanner.nextToken();
       if (!Character.isWhitespace(aWord.charAt(0))) {
          sb.append(aWord + "\n");
       }
    }
    return sb.toString();
 }//end tokenizeEx

Example Applet

The following applet uses the above method. TokenizeExpr.java. Unfortunately, most browsers don't run Java 2 applets correctly yet, so it's doubtful that you can see it working here. You may have to compile and run it yourself.