Here are three sets of data/expected results for testing the Tokenizer.
Each set consists of a Core program followed by the expected results; 
the expected results in each case is a sequence of integers, these being
the token numbers corresponding to the tokens in the given Core program.
[Note that your tokenizer should *not* assume that the input will be a 
Core program, only that it is a sequence of tokens; moreover, if your
tokenizer comes across something that is not a legal token, it should 
print a suitable error message and stop. But the test data below are all
legal Core programs; you can use them also as test data for the later 
portions of the Core Interpreter.]

Note that the expected outputs shown below are written with multiple 
numbers on each line; in your actual Tokenizer, there should be one 
number per line. Make sure the token numbers you generate match the ones
specified in the lab assignment. So the eleven reserved words:
  program, begin, end, int, if, then, else, while, loop, read, write
have token numbers 1 through 11 respectively. 

The ninteen special symbols:
  ; , = ! [ ] && || ( ) + - * != == < > <= >=
have token numbers 12 through 30. Unsigned integers have token number 31.
Identifiers have token number 32. And end-of-file has token number 33.

If you find mistakes in either the inputs or the expected results, please
email or post. 

--Neelam.


===========
program
 int X ;
 begin X = 25 ; write X ; end

Expected output:
 1 4 32 12 2 32 14 31 12 11 32 12 3 33

(note: 33 is the "EOF" token).
===========

===========
program int ABC, D;

          begin read ABC; read D;

               while (ABC != D) loop
                                     if (ABC > D) then ABC = ABC - D;

                                     else D = D - ABC;

                                end;

                            end;

               write D;

          end

Expected output:
1 4 32 13 32 12 2 10 32 12 10 32 12 8 20 32 25 32 21 9 5 20 32 28 32 21 
6 32 14 32 23 32 12 7 32 14 32 23 32 12 3 12 3 12 11 32 12 3 33
===========

===========
program int X1, X4; int X2, X3, X7;

          begin X1=0; X2=1; X3=1; read X4;

               while (X1 < X4) loop

                                X7=X2+X3; X2=X3;

                                X3=X7; X1=X1+1;

                            end ;

               write X2;

          end


Expected output:
1 4 32 13 32 12 4 32 13 32 13 32 12
2 32 14 31 12 32 14 31 12 32 14 31 12 10 32 12
8 20 32 27 32 21 9 32 14 32 22 32 12 32 14 32 12
32 14 32 12 32 14 32 22 31 12 3 12 11 32 12 3 33