nginx hotlink protection for missing referer, but allow direct URL request

Question

I'm using the following hotlink protection for Nginx I found from this page:

https://nixcp.com/anti-hotlinking-nginx/

I use this code:

location ~* \.(gif|jpg|jpeg|png|webp|svg|ico)$ {

valid_referers none blocked mysite.com ~.mysite.com server_names ~($host);
    if ($invalid_referer) {
        return  403;
    } 
}

When I remove "none" from the valid_referers to prevent pages that hotlink without referer, I noticed this also affects direct URL requests from browser URL bar.

My question is: is there any way to block websites that hotlink and send requests with the “Referer” field is missing in the request header; but at the same time allow direct URL requests from the browser?

Thanks!