COP 4020 Programming Assignment 3: Scanning a C++ Program

Educational Objectives: After completing this assignment, the student should be able to do the following:

Operational Objectives: Use a scanner to count the number of identifiers and unique identifiers in an arbitrary C++ program, excluding C++ key words and comments.

Deliverables: One file Pr3.java

For this assignment we assume you are familiar with C++ or Java. This assignment contains Java code, but you don't need to be an experienced Java programmer to implement the assignment.

A scanner breaks up a character stream into tokens. The purpose of scanning is to simplify the task of the parser of a compiler. Scanners are also used by all sorts of software components that have to deal with the interpretation of documents (e.g. parsers for SQL, HTML, XML and various text and data processors).

The following Java Scanner program reads a file sequentially, tokenizes the input data, and sends the resulting token sequence to standard output:

/* Scanner.java
   Implements a simple lexical analyzer based on java.io.StreamTokenizer

   Compile:
   javac Scanner.java

   Execute:
   java Scanner 
*/

import java.io.*;

public class Scanner
{ public static void main(String argv[]) throws IOException
  { InputStreamReader reader;
    if (argv.length > 0)
      reader = new InputStreamReader(new FileInputStream(argv[0]));
    else
      reader = new InputStreamReader(System.in);

    // create the tokenizer:
    StreamTokenizer tokens = new StreamTokenizer(reader);
    tokens.ordinaryChar('.');
    tokens.ordinaryChar('-');
    tokens.ordinaryChar('/');

    // keep current token in variable "next":
    int next;

    // while more input, split input stream up into tokens and display:
    while ((next = tokens.nextToken()) != tokens.TT_EOF)
    { if (next == tokens.TT_WORD)
      { System.out.println("WORD:   " + tokens.sval);
        break;
      }
      else if (next == tokens.TT_NUMBER)
      { System.out.println("NUMBER: " + tokens.nval);
        break;
      }
      else
      { switch ((char)next)
        { case '"':
            System.out.println("STRING: " + tokens.sval);
            break;
          case '\'':
            System.out.println("CHAR:   " + tokens.sval);
            break;
          default:
            System.out.println("PUNCT:  " + (char)next);
        }
      }
    }
  }
}

Copy (and download if needed) this example scanner Java program from

~cop4020p/fall08/examples/Scanner.java

Compile it (e.g. on the linprog stack):

javac Scanner.java

To test it, execute it to scan the source file itself:

java Scanner Scanner.java

The number of source code lines is often used as a measure of the complexity of a program. Another measure of program complexity is to find the number of distinct identifiers used in a program source code, such as variable names, class names, method and function names, and so on.

Based on the example Java Scanner program, write a Java program that counts the total number of identifiers and the number of distinct identifiers of a C++ program source file. Your scanner will have to ignore comments for a fair measure. Otherwise, the words of the comments will count as identifiers.

You also need to ignore keywords in the source input to the scanner. To ignore keywords, you can compare the value of a token to the string representations of C++ keywords. For example, tokens.sval.equals("return") is 'true' if the current token is the "return" keyword, which should not be counted. Find a textbook on C++ and write down all C++ keywords (alphanumeric keywords). These need to be ignored in the count.

To count the number of distinct identifiers, you also need a set data structure to keep track of the occurrences of the identifier names. For this, you can use the Java fragment below which illustrates the use of the java.util.HashSet class:

import java.util.*; // import java.util.HashSet class
  ...
    HashSet<String> idset = new HashSet<String>(); // create a new HashSet instance
    ...
    String s;         // to hold an identifier name from input
    ...
    idset.contains(s) // is true if s is a key in the set
    idset.add(s)      // add a key s to the set
    idset.size()      // returns the size of the set
    ...

More details about Java packages and classes such as java.util.HashSet can be found at:

http://java.sun.com/j2se/1.4.2/docs/api/index.html

More details about the Java tokenizer can be found at:

http://java.sun.com/j2se/1.4.2/docs/api/java/io/StreamTokenizer.html

Name your new program file Pr3.java. Note that the program class in this file must be named Pr3 as well to compile. Your program should print the total number of identifiers that occurs in the source input and the number of distinct identifiers. For example:

javac Pr3.java
java Pr3 myfile.cpp
Total number of identifiers = 37
Distinct identifiers = 19

Turn in this assignment as a working Java program named "Pr3.java", using the submit script "pr3submit.sh".

Hint: