To add a custom restriction message for content that is restricted by Group or Member Type, use the rcpbp_restricted_message filter located in includes/restricted-content.php.

An example can be found here.

Parameters:

$message - string, restricted content message.
$is_group_restricted - boolean, shows if this content has an unmet group restriction.
$is_member_type_restricted - boolean, shows if this content has an unmet member type restriction.