How to restrict Admin Tools by IP?

[Rajinsharwar]

I am trying to enable IP restriction for ee-admin/*, any idea how can I do that? I already tried the below in my-site/config/nginx/custom/admin-tools.conf, but it seems due to nginx proxy, I am unable to capture the actual IP of the visitor here.

location ^~ /ee-admin/ {
    root /var/www/htdocs;

    allow MY_IP_HERE;
    deny all;

    location ~ \.php$ {
        try_files $uri =404;
        include fastcgi_params;
        fastcgi_pass php;
    }
}