As a subtab of in Service cloud console in Salesforce

"Subtabs of" in Salesforce Lightning and "As a subtab of " in Salesforce Classic is used to decide how records display when they are selected from outside of primary tabs or subtabs.

For example, when users select cases from list views, search results or screen pops, you can select cases to display as primary tabs or as subtabs on parent records, such as accounts.
 
Salesforce Lightning:


Salesforce Classic:


In the above example, for Case I have selected as "As subtab of Account Name(Account)".

Sample Case in Account:


After opening the case: