To display checkbox options in vertical direction, layout property is used.

Example:

<apex:selectCheckboxes value=”{!relObjs}” layout=”pageDirection”>
    <apex:selectOptions value=”{!relObj}”/>
</apex:selectCheckboxes>