As you surmise, and confirmed in comments, you can't do this with standard Linux file permissions without granting some access to the parent directory.
You can do this using Apache (.htaccess) directives. However, it's not clear from your question whether you need public access to all "sub-directories inside it", or only the specific subdirectory "/uploads/secure/public", as used in your example. These are best resolved different solutions.
Whilst this can be done using client-side .htaccess files, it would be preferable to use <Directory> containers in the main server config. (Although your use of the cpanel tag does suggest a "shared" server, so that is probably not possible in your case.)
Public access to only a single subdirectory:
In the /uploads/secure/.htaccess file, deny "public" access to this directory and all subdirectories:
# /uploads/secure/.htaccess
Require all denied
In the subdirectory you wish to allow public access then add the following to "override" the above:
# /uploads/secure/public/.htaccess
Require all granted
All other subdirectories are denied access, unless you create an additional .htaccess for each one (cumbersome to maintain if there are many).
Public access for all subdirectories:
If you need to protect only the parent directory and allow access to all subdirectories then you could use an Apache expression (requires Apache 2.4) to target the URL-path of the parent directory (and all files within) only. For example, in the parent .htaccess file:
# /uploads/secure/.htaccess
<If "%{REQUEST_URI} =~ m#^(/[^/]+){2}/[^/]*$#">
Require all denied
</If>
The regex ^/[^/]+/[^/]+/[^/]*$ only matches URLs of the form /<dir>/<dir>/ (eg. only /uploads/secure/ would apply since the .htaccess file is located in that directory) or /<dir>/<dir>/<anyfile>, so the restriction does not apply to URLs of the form /upload/secure/<subdir>/<file>.
(This is more easily achieved using <Directory> containers in the server config.)
Although if you are using .htaccess files then you might choose to maintain just a single .htaccess file in the document root, in which case you would need to make the regex more specific. For example:
# /.htaccess
<If "%{REQUEST_URI} =~ m#^/uploads/secure/[^/]*$#">
: