Write the function GetToken: bool GetToken(char Block[], char Token[], int &Toke

Question

Write the function GetToken: bool GetToken(char Block[], char Token[], int &TokenType;, &pos;); This function takes a block of characters stored in Block[] and copies the next 'token' into the Token[] array. TokenType will store the type of token this is: 1: Words, 2: Numbers, 3: Punctuation, etc. In order to test this function, write functions to extract a block of text from a text file storing it in the Block[] array. Then, call on GetToken in a loop to extract successive tokens form the Block array and printing the returned token and its type in each line. Block[] : "This is the 3rd time I am typing this. lol!!" output: This [1] is [1] the [1] 3 [2] rd [1] time [1] I [1] am [1] typing [1] this [1] . [3] lol [1] ! [3] ! [3] Design: As you design this function, keep in mind that the requirements for what constitutes a Token could change easily over time and you should not have to redesign your function: You cannot be looking at a single character and decide if this character is part of a given token. Take the character '2'. As of now, '2' is part of a Numerical token, like "423" but what if we decided that our 'words' could contain numbers: "R2D2"? Think about this for a second... Here is what I propose: The pre-conditions for the GetToken function is that Block[] contains a block of text you are trying to tokenize. and pos is the index of the first character of the next token to be extracted. The post-conditions are that Token[] contains the next token, TokenType contains a code corresponding to the type of token extracted. ie 1 for words, 2 for numbers, etc. pos contains (points to) the first character of the next token to be extracted. and Block[] Stays the same. The return value of the function is a bool. It is true if pos has reached to the end of Block[] and false otherwise. Now... Your GetToken function performs 5 steps: Step 0: Push pos to the beginning of the next token if it is not already there. This is useful if the Bock contains extra spaces: " I have blank spaces " Step1: Examine the character pointed to by pos (the first character in the token) and determine its type. This will need its own function, of course. Step2: a switch statement that will call different functions such as GetWord, GetNumber, GetPunc that will be passed the Block[], the Token[], and the pos. Each function is specialized in extracting a specific type of token. By the end of this step, Token[] will contain the next token and the TokenType is set to the appropriate value. Step 3: Push pos to the start of the next token so that it is ready for the next time the function is called. Step4: Decide if you have reached the end of the Bock[] and return true if you have and false otherwise. Make sure your function is properly written and is robust and versatile. You will be using this function for many semesters to come. Here and where you transfer. One last note: BE WARE OF THE DREADED "OFF BY ONE" ERRORS. These are very hard to find and devastating to your function.

Explanation / Answer

/* pupose: picking the character from the chunk of text, then analyst what type of that token is */ #include #include #include #include using namespace std; const int max = 1000; int type(char block[],int pos); // type of the token void getword(char block[], char token[], int &pos); void getnumber(char block[], char token[], int &pos); void print(char token[],int type); void getpunc(char block[], char token[], int &pos); bool gettoken(char block[],char token[], int &type,int &pos); int main() { int type; int pos = 0; char block[1000]= "Writing code is extremely hard!"; system("PAUSE"); return 0; } int type(char block[],int pos) { if(isalpha(block[pos])) return 1; if(isalnum(block[pos])) return 2; if(ispunct(block[pos])) return 3; } //Realize the word is the type void getword(char block[], char token[], int &pos) { int i, type; while(isalpha(block[pos])) { token[i]=block[pos]; i++; pos++; token[i] = '\0'; } } //Realize the number is the type void getnumber(char block[], char token[], int &pos) { int i,type; while(isalnum(block[pos])) { block[pos]= token [i]; i++; pos++; token[i]='\0'; } } //Realize the punctuation is the type void getpunc(char block[], char token[], int &pos) { int i, type; while(ispunct(block[pos])) { block[pos] = token[i]; i++; pos++; token[i]='\0'; } } bool gettoken(char block[],char token[], int &type,int &pos) { token[0] = block[pos]; token[1] = '\0'; type = 1; pos++; if ( pos >strlen(block)) return true; else return false; } void print(char token[],int type) { for(int i = 0; i

Related Questions

Navigate

• Browse (All)
• Browse W
• Subjects

---

---

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.