What are Tokens in programming
What are Tokens in programming
Smallest individual element of a program is called as Token. Everything you see inside a program is a token.
For example - Suppose an English sentence. "C language is an awesome language. C was developed by Dennis Ritchie at AT&T Bell labs in 1972."
The above sentence is made of Alphabets (a-z A-Z), Blank spaces, Digits (0-9) and special characters (full stop in our case). These are building blocks or basic elements of our sentence. Similarly there are various basic programming elements that makes any program.
There are five types of tokens.
1.Keyword
2.Identifier
3.Operator
4.Separator
5.Literal
What are Keywords?
Keyword is a reserved word whose meaning is already defined by the programming language. We cannot use keyword for any other purpose inside programming. Every programming language have some set of keywords. Examples: int, do, while, void, return etc(Note: These keywords are common to C and C influenced languages).
What are Identifiers?
Identifiers are the name given to different programming elements. Either name given to a variable or a function or any other programming element, all follow some basic naming conventions listed below:
1.Keywords must not be used as an identifier.
2.Identifier must begin with an alphabet a-z A-Z or an underscore_ symbol.
3.Identifier can contains alphabets a-z A-Z, digits 0-9 and underscore _ symbol.
4.Identifier must not contain any special character (e.g. !@$*.'[] etc.) except underscore _.
Examples of some valid identifiers
num, Num, _num, _Num, num1, Num1, _num1, _Num1, _1num, _1Num, _num_, number_to_add
etc.
Examples of some invalid identifiers
1num, number to add, 1_num, num-to-add, num@
etc.
What are Operator?
Operators are the symbol given to any
arithmetical or logical operations. Various programming languages provides various sets of operators some common operators are:Lets suppose two variables a=10, b=5
- operator subtracts second operand from first i.e. a - b and results 5.
| Operator | Description | Example |
|---|---|---|
| Arithmetic operator | ||
| Arithmetic operator are used to perform basic arithmetic operations. | ||
| + | Adds two operand. | a + b gives 15 |
| * | Multiplies two operands. | a * b gives 50 |
| / | Divides two operands. | a / b gives 2 |
| % | Modulus operator divides the first operand from second and returns the remainder. It is generally used for checking divisibility. | a % b gives 0 (As 10/5 will have 0 remainder) |
Assignment operator
Assignment operator is used to assign value to a variable. The value is assigned from right to left.
| = | Assigns value from right operand to left operand. | a = 10 will assign 10 in a |
Relational operator
Relational operator are used to check relation between any two operands. Whether any of them is greater, equal or not equal.
| > | If value of left operand is greater than right, returns true else returns false | (a > b) will return true |
|---|---|---|
| < | If value of right operand is greater than left, returns true else returns false | (a < b) will return false |
| == | If both operands are equal returns true else false | (a == b) will return false |
| != | If both operands are not equal returns true else false. | (a != b) will return true |
| >= | If value of left operand is greater or equal to right operand, returns true else false | (a >= b) will return true |
| <= | If value of right operand is greater or equal to left operand, returns true else false | (a <= b) will return false |
Logical operator
Logical operator are used to combine two boolean expression together and results in a single boolean value according to the operand and operator used.
| && | Used to combine two expressions. If both operands are true or Non-Zero, returns true else false | ((a>=1) && (a<=10)) will return true since (a>=1) is true and also (a<=10) is true. |
|---|---|---|
| || | If any of the operand is true or Non-zero, returns true else false | ((a>1) || (a<5)) will return true. As (a>1) is true. Since first operand is true hence there is no need to check for second operand. |
| ! | Logical NOT operator is a unary operator. Returns the complement of the boolean value. | !(a>1) will return false. Since (a>1) is true hence its complement is false. |
Bitwise operator
Bitwise operator performs operations on Bits(Binary level). Lets suppose a = 10, b = 5
a = 0000 1010 (8-bit binary representation of 10)
b = 0000 0101 (8-bit binary representation of 5)
| & | Bitwise AND performs anding operation on two binary bits value. If both the values are 1 then will result is 1 else will result in 0. | 0000 1010 & 0000 0101 ____________ 0000 0000 |
|---|---|---|
| | | Bitwise OR returns 1 if any of the two binary bits are 1 else returns 0. | 0000 1010 | 0000 0101 ___________ 0000 1111 |
| ^ | Bitwise XOR returns 1 if both the binary bits are different else returns 0. | 0000 1010 ^ 0000 0101 ___________ 0000 1111 |
| ~ | Bitwise COMPLEMENT is a unary operator.It returns the complement of the binary value i.e. if the binary bit is 0 returns 1 else returns 0. | ~ 0000 1010 ___________ 1111 0101 |
| << | Bitwise LEFT SHIFT operator is also unary operator. It shift the binary bits to the left. It inserts a 0 bit value to the extreme right of the binary value. Or we may say it generally multiplies the value with 2. | 0000 1010 << 2 = 0010 1000 |
| >> | Bitwise RIGHT SHIFT operator is an unary operator. It shifts the binary bits to the right. It inserts a 0 bit value to the extreme left of the binary value. Or we may say it generally divides the value with 2. | 0000 1010 << 2 = 0000 0010 |
Increment/Decrement operator
Increment/Decrement operator is a unary operator used to increase an integer value by 1 or decrease it by 1. Increment/decrement operator are of two types Postfix and Prefix.
| ++ | Increment operator will add 1 to an integer value. | a++ will give 11 ++a will also give 11 |
|---|---|---|
| -- | Decrement operator will subtract 1 from an integer value. | a-- will give 9 --a will also give 9 |
Conditional/Ternary operator
Ternary operator as a conditional operator and is similar to simple if-else. It takes three operand.
| ?: | It is used as conditional operator. Syntax of using ternary operator: (condition) ? (true part) : (false part) | b = (a>1) ? a : b;will store the value 10 in b as (a>1) is true hence true part will execute, assigning the value of a in b. |
What are Separators?
Separators are used to separate different programming elements. The various types of separators used in programming are:
(Space) \t(Tab) \n(New line) . , ; () {} []
What are Literals?
Literals are constant values that are used for performing various operations and calculations. There are basically three types of literals:
1.Integer literal
An integer literal represents integer or numeric values.
Example: 1, 100, -12312 etc
2.Floating point literal
Floating point literal represents fractional values.
Example: 2.123, 1.02, -2.33, 13e54, -23.3 etc
3.Character literal
Character literal represent character values. Single character are enclosed in a single quote(' ') while sequence of character are enclosed in double quotes(" ")
Example: 'a', 'n', "Hello", "Hello123" etc.
