When the users upload files, it will be stored in ContentDocument Entity. So, develop a trigger on ContentDocument entity to fire when the users upload files in Salesforce.

Sample Trigger:

trigger ContentDocumentTrigger on ContentDocument ( before insert ) {

    for ( ContentDocument objCD : trigger.new ) {

        if ( objCD.Title == ‘test’ ) {

            objCD.addError( ‘Do not upload Test File’ );

        }
    }

}

Output: