Package components.utilities

Class Tokenizer

java.lang.Object
components.utilities.Tokenizer
public final class Tokenizer extends Object
Tokenizer utility class with methods to tokenize an input stream and to perform various checks on tokens.

  • Field Details

    • END_OF_INPUT

      public static final String END_OF_INPUT
      Token to mark the end of the input. This token cannot come from the input stream because it contains whitespace.
      See Also:

  • Method Details

    • tokens

      public static Queue<String> tokens(SimpleReader in)
      Tokenizes the entire input getting rid of all whitespace separators and returning the non-separator tokens in a Queue<String>.
      Parameters:
      in - the input stream
      Returns:
      the queue of tokens
      Updates:
      in.content
      Requires:
      in.is_open
      Ensures:
      tokens =
  [the non-whitespace tokens in #in.content] * <END_OF_INPUT>  and
in.content = <>

    • isKeyword

      public static boolean isKeyword(String s)
      Reports whether the given String is a keyword in the BL language.
      Parameters:
      s - the String to check
      Returns:
      true if s is a keyword, false otherwise
      Ensures:
      isKeyword = [s is a keyword in the BL language]

    • isCondition

      public static boolean isCondition(String s)
      Reports whether the given String is a condition in the BL language.
      Parameters:
      s - the String to check
      Returns:
      true if s is a condition, false otherwise
      Ensures:
      isCondition = [s is a condition in the BL language]

    • isIdentifier

      public static boolean isIdentifier(String s)
      Reports whether the given String is an identifier in the BL language.
      Parameters:
      s - the String to check
      Returns:
      true if s is an identifier, false otherwise
      Ensures:
      isIdentifier = [s is an identifier in the BL language]