Skip to content

DNS New Features: disableCache, finalQuery, unexpectedIPs, "*", UseSystem-queryStrategy, useSystemHosts #4666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

patterniha
Copy link
Contributor

@patterniha patterniha commented Apr 29, 2025

New features:

disableCache for each DNS-Server-Object:

currently we have only one global disableCache option that affects all DNS-servers, but we may want to disable the cache only for a specific DNS-Server.

///

finalQuery for DNS-Server-Object:**

Suppose you want to use DNS-Server-A for "youtube.com", but use DNS-Server-B for other google sites and use DNS-Server-C for others, so you should set:

"dns":{
    "servers": [
      {
       "address": "server-A",
       "domains": ["youtube.com"]
      },
      {
       "address": "server-B",
       "domains": ["geosite:google"]
      },
      "server-C"
    ]
}

But for whatever reason, server-A may be unavailable for a while(for example, the network may be unreachable for a while) so it uses server-B for "youtube.com", but we don't want this to happen.

Currently, there is no mechanism to prevent using server-B for "youtube.com", this is due to strange behavior of skipFallback (except creating custom-geosite where "youtube.com" is removed from "google" list, but this is not possible for all users)

but now we can set finalQuery= true for server-A, so any result from server-A return as a final-result and no other DNS-server will be performed.

"dns":{
    "servers": [
      {
       "address": "server-A",
       "domains": ["youtube.com"],
       "finalQuery": true
      },
      {
       "address": "server-B",
       "domains": ["geosite:google"]
      },
      "server-C"
    ]
}

///

unexpectedIPs for DNS-Server-Object:**

Suppose we want no IP to be in an IP-range-A, and if all IPs in IP-range-A, the next-dns-fallback performed.
for example for Serverless-for-Iran anti-sanction-version, i want to use a anti-sanction DNS, but goverment-run-anti-sanction-DNS only bypass sanctions and not filter.
IRGFW return 10.10.34.0/24, 2001:4188:2:600:10:10:34:0/120 range for blocked domain, so if the return-IPs is in these ranges, the fallback-DNS should be performed.

one way to achieve this goal is creating custom-geosite and then using ! sign, but this is not possible for all users.
another way is to calculate reverse-CIDR-list, for example using online-tools to calculate reverse-CIDR-list, but the reverse-CIDR-list is long and it causes the configuration to be messy.

as a result i add unexpectedIPs option, and an IP is matched if and only if does not match any of the IP-ranges in the unexpectedIPs list, in other words:
expectedIPs = [0.0.0.0/0, ::/0] - unexpectedIPs.

also, we may need all IPs to be in range-A, and no IP to be in range B, so we need both expectedIPS and unexpectedIPs: expectedIPS=[range-A], unexpectedIPs=[range-B]

"dns":{
    "servers": [
      {
       "address": "anti-sanction-dns",
       "unexpectedIPs": ["10.10.34.0/24", "2001:4188:2:600:10:10:34:0/120"]
      },
      "fallback-dns"
    ]
}

///

Add to Documentation:

(dns-server settings)

disableCache: bool

* The cache is disabled, if disableCache = true in DNS-settings or this DNS-Server-settings.

///

finalQuery:bool

* when true, the result is returned in any case(even when IP-list is empty) and no other fallback will be performed.

///

expectedIPs: [string]

...

if you add "*" in this list, the original-IPs still returned if no ip matched.


///
unexpectedIPs: [string]

* reverse form of expectedIPs, an IP is matched if and only if does not match any of the IP-ranges in the list, in other words: 
`expectedIPs = [0.0.0.0/0, ::/0] - unexpectedIPs.`

if you add "*" in this list, the original-IPs still returned if no ip matched.

///
queryStrategy: "UseIP" | "UseIPv4" | "UseIPv6" | "UseSystem"

when queryStrategy is UseSystem, every time dns-Query call, it check the system-network to see if it supports IPv6(and IPv4) or not, if it support IPv6(or IPv4), the IPv6(or IPv4) is also returned, otherwise not returned.

///
(dns settings)

useSystemHosts: bool

if true, system-hosts appends to config-hosts at start, default is false.

@j2rong4cn
Copy link
Contributor

@RPRX 我写了个只返回rcode的DNS服务,要不要合并到这里?

{
  "domains": [
    "geosite:category-ads-all",
    "domain:lan",
    "regexp:^[^\\.]*$"
  ],
  "skipFallback": true,
  "finalQuery": true,
  "address": "rcode://3"
}

j2rong4cn@d08f648

@patterniha
Copy link
Contributor Author

patterniha commented Apr 29, 2025

@j2rong4cn

although, it is possible to add rcode-only-dns-server but it is better to add this feature to the DNS-hosts instead of adding rcode-only-dns-server.

i change the hosts code to achieve this goal, but currently we only have rcode=0: #4673

there is no difference between rcode=0, and rcode=3 or others in practice, because we don't return any IP in any case.

anyway, if rcode-number is important for your job, you can extend hosts code to return your desired rcode.

anyway, @RPRX has to make the final decision and I just said my opinion.

@j2rong4cn
Copy link
Contributor

@patterniha
I need to return rcode=3. I used nslookup to test and found that nslookup will fallback when rcode=0.
I see StaticHosts.Lookup does not return an error, so I wrote rcode-only-dns-server

@patterniha
Copy link
Contributor Author

patterniha commented Apr 29, 2025

@patterniha I need to return rcode=3. I used nslookup to test and found that nslookup will fallback when rcode=0. I see StaticHosts.Lookup does not return an error, so I wrote rcode-only-dns-server

so i think it is better to extend hosts code, for example:

"geosite:category-ads-all": "#3"

This is definitely a better and more direct way, anyway @RPRX gives the final opinion.

@j2rong4cn
Copy link
Contributor

@patterniha I need to return rcode=3. I used nslookup to test and found that nslookup will fallback when rcode=0. I see StaticHosts.Lookup does not return an error, so I wrote rcode-only-dns-server

so i think it is better to extend hosts code, for example:

"geosite:category-ads-all": "#3"

This is definitely a better and more direct way, anyway @RPRX gives the final opinion.

Good, let's do as you say.

@RPRX

This comment was marked as off-topic.

@patterniha
Copy link
Contributor Author

patterniha commented Apr 30, 2025

After a quick look, it is not feasible to geoipchange directly to this breaking change, which will seriously damage the existing configuration.expected_geoip

@RPRX

No, it is 100% compatible with current configuration.

it does not change any current logic and configuration.

I just move codes to "condition_geoip.go", because we need to have more general function for unexpectedIPs, priorIPs and similar needs in the future.

It seems like you looked too quickly.

///

The only break is allowUnexpectedIPs and I don't think it has been used and it can be safely changed to priorIPs.

@gfw-killer
Copy link

Thank you, as you are focused on Xray DNS, please have a look at this Issue too #4677

@j2rong4cn
Copy link
Contributor

so i think it is better to extend hosts code, for example:

"geosite:category-ads-all": "#3"

Check this out
j2rong4cn@43b71df

@patterniha
Copy link
Contributor Author

so i think it is better to extend hosts code, for example:
"geosite:category-ads-all": "#3"

Check this out j2rong4cn@43b71df

you should open a PR, I'm just a contributor like you.

@RPRX
Copy link
Member

RPRX commented Apr 30, 2025

我觉得用 priorIPs 取代 allowUnexpectedIPs 是合理的,不过 geo 是支持反选的比如 "geoip:!cn",所以两个 un 似乎没必要

@RPRX
Copy link
Member

RPRX commented Apr 30, 2025

不过现在反选只支持 geo,如果有需要的话再加一个反选 IP/CIDR 的语法就差不多了

@RPRX
Copy link
Member

RPRX commented Apr 30, 2025

还有就是先 expectedIPspriorIPs,看名称的话顺序不太对,应当把 priorIPs 改名为 secondExpectedIPs

@patterniha
Copy link
Contributor Author

I think it is reasonable to priorIPsreplace with allowUnexpectedIPs, but geo supports inverse selection "geoip:!cn", so two uns seem unnecessary.

@RPRX

Yes, support inverse but:

  1. You cannot and two inverse, for example you cannot have !geoip:cloudflare && !geoip:cloudfront
  2. You can inverse only geoip, and you cannot inverse Non-geoip cird, for example i need inverse of {10.10.34.0/24 + 2001:4188:2:600:10:10:34:0/120} for serverless-for-Iran-anti-saction but we can't even have inverse of {10.10.34.0/24}, let alone the inverse of both.

@patterniha
Copy link
Contributor Author

Also, first , expectedIPsthen priorIPs, the order is not right if you look at the names, you should priorIPschange tosecondExpectedIPs

First, We will rarely need both at the same time.
Second, Even if we have both, it doesn't matter whether we apply expectIPs first or priorIPs first.

@RPRX
Copy link
Member

RPRX commented May 1, 2025

First, We will rarely need both at the same time.
Second, Even if we have both, it doesn't matter whether we apply expectIPs first or priorIPs first.

这俩不就是为了配合起来使用的吗,expect 两次,且有一个明确在前面,还有人要的话就 third

@RPRX
Copy link
Member

RPRX commented May 1, 2025

话说我突然想到,expectedIPs 作为一个数组,可能也像 #4673 以前的 hosts 一样把内容给合并了

或许直接更改它的行为,改为不合并,确保数组内第一个就是 first,第二个就是 second,以此类推,还是说本来就是这样的逻辑

@patterniha
Copy link
Contributor Author

ExpectedIPs, unexpectedIPs, priorIPs, unpriorIPs
Are 4 distinct options, Although we can have several of them at the same time,but it is a rarely used.

ExpectedIPs is clear.

unexpectedIPs is reverse-form of ExpectedIPs and i explained why we need reverse-form too.

priorIPs logic is simple: act same as expectedIPs but if no IP matched, the original-IPs still returned without any change, it is useful for workers/MitM configs.

unpriorIPs is reverse-form of priorIPs

@RPRX
Copy link
Member

RPRX commented May 1, 2025

我误解了你,你也误解了我,我本来以为你那个 priorIPs 是填 "0.0.0.0/0" 和 "::/0" 用的,作为第二层 expected

但是这个误解正好导致一个更简洁而强大的想法:把 expectedIPs 改为逐级匹配的,这一点你还没看懂 #4666 (comment)

至于如何合并反选,我还得再想想

@patterniha
Copy link
Contributor Author

It suddenly occurred to me that expectedIPsas an array, it might be possible to merge the contents like before #4673hosts

Maybe just change its behavior to not merge, making sure the first one in the array is first, the second one is second, and so on.Or is this the logic?

Suppose you want your IPs to be in range A but not in range B.
So you simply set:
ExpectedIPs = [A], unexpectedIPs = [B]

Therefore, the presence of unexpectedIPs next to expectedIPs is mandatory and the goal cannot be achieved by just changing syntax.

@RPRX
Copy link
Member

RPRX commented May 1, 2025

我觉得四个选项叠一起的话逻辑有点混乱,压成两个选项就行,unexpectedIPs 是可以加的,它们是合并而不是逐级,且作用于 expectedIPs 每一级匹配成功之后(除非 "0.0.0.0/0 或 "::/0"),比如第一级匹配到了但被 unexpectedIPs 否决了就继续下一级

@RPRX
Copy link
Member

RPRX commented May 1, 2025

这样就可以 cover 所有需求了,更少的配置项,更强大的功能

@patterniha
Copy link
Contributor Author

As i understand, you agree with unexpectedIPs but not with priorIPs and unpriorIPs.

So what is alternative for priorIPs = [geoip:cloudflare] in your method?

@RPRX
Copy link
Member

RPRX commented May 1, 2025

比如如下配置

"expectedIPs": ["geoip:us", "geoip:uk", "0.0.0.0/0", "::/0"],
"unexpectedIPs": ["10.10.34.0/24", "2001:4188:2:600:10:10:34:0/120"]

逻辑是

  1. expect US IPs, if we picked an IP, check all unexpectedIPs, if no IP was picked or the picked IP is "unexpected" then
  2. expect UK IPs, if we picked an IP, check all unexpectedIPs, if no IP was picked or the picked IP is "unexpected" then
  3. allow all IPv4, do not check unexpectedIPs
  4. allow all IPv6, do not check unexpectedIPs

@patterniha
Copy link
Contributor Author

  1. This break current expectedIPs behavior.

    If the IPs contain both uk and us, now we only have us ips.

  2. Many changes need to be made in the code.

  3. It is definitely more complicated than using 4 options

@RPRX
Copy link
Member

RPRX commented May 1, 2025

这并不会严重破坏 expectedIPs 的行为,只是把 US 放第一级、UK 放第二级了,这可以接受,用户配置里本来就是这个顺序,我不确定现有的代码逻辑里是合并了还是逐级匹配,但是改成逐级匹配没有难度,就只是改成 for 循环,比多两个选项要简单

并且如果有人要 third 或者更多,他也可以通过优先级顺序来实现

@RPRX
Copy link
Member

RPRX commented May 1, 2025

简单来说就是你的 PR 在广义范围上的 expected 功能只有两级且第二级只能全选,我提的是 expected 多级且每级范围可以自定义

@RPRX
Copy link
Member

RPRX commented May 1, 2025

@j2rong4cn 我刚看到 #4666 (comment) ,可以开个 PR

@RPRX
Copy link
Member

RPRX commented May 1, 2025

@patterniha 这个 PR 加的 unexpectedIPs 和删掉的 allowUnexpectedIPs 也是围绕 "expected" 服务的,所以就改这个 PR 吧

@RPRX
Copy link
Member

RPRX commented May 1, 2025

倒是可以把 disableCachefinalQuery 这俩和 "expected" 不相关的分两个 PR 出去

@patterniha
Copy link
Contributor Author

For example, the following configuration

"expectedIPs": ["geoip:us", "geoip:uk", "0.0.0.0/0", "::/0"],
"unexpectedIPs": ["10.10.34.0/24", "2001:4188:2:600:10:10:34:0/120"]

The logic is

  1. expect US IPs, if we picked an IP, check all unexpectedIPs, if no IP was picked or the picked IP is "unexpected" then
  2. expect UK IPs, if we picked an IP, check all unexpectedIPs, if no IP was picked or the picked IP is "unexpected" then
  3. allow all IPv4, do not check unexpectedIPs
  4. allow all IPv6, do not check unexpectedIPs

@RPRX
There are a few problems:

  1. You seperate 0.0.0.0/0 and ::/0 behavior from other ranges behavior.
  2. We can't have both IPv4 and IPv6, and only one of them can be returned, This is a serious problem for happy-eyeballs.

@RPRX
Copy link
Member

RPRX commented May 1, 2025

You seperate 0.0.0.0/0 and ::/0 behavior from other ranges behavior.

这俩就是兜底的,前面所有流程走完了还没选定 IP 就返回第一个 IP,所以不适用 unexpectedIPs

We can't have both IPv4 and IPv6, and only one of them can be returned, This is a serious problem for happy-eyeballs.

A 查询和 AAAA 查询本来就指定了 IP 类型,这里两种都写只是方便这个 DNS Server 配置可以同时 cover 两种查询

@patterniha
Copy link
Contributor Author

patterniha commented May 1, 2025

A 查询和 AAAA 查询本来就指定了 IP 类型,这里两种都写只是方便这个 DNS Server 配置可以同时 cover 两种查询

For client/browser dns-query we have two distinct request for IPv4 and IPv6(pass through dns-proxy) so your changes does not affect this case .

But for happy-eyeballs we use domainStrategy= ForceIP and we have one merged request for IPv4 and IPv6 but with your changes only one type of IPs is returned.

DNS behaves differently when we have two separate requests or one merged request.

@RPRX
Copy link
Member

RPRX commented May 1, 2025

那就改成一个简单的语法糖:, "any"

@RPRX
Copy link
Member

RPRX commented May 1, 2025

改成 , "*" 吧,且它不适用 unexpectedIPs

* 比 any 看起来更规范一些

@patterniha
Copy link
Contributor Author

patterniha commented May 1, 2025

@RPRX

expectedIPs and unexpectedIPs should behave similarly.

Also, if we have expectedIPs = [ipv4-cidr, ipv6-cidr] only ipv4-cidr is returned.

so In completion of your word:

Let's make expectedIPs and unexpectedIPs two separate options.

if "*" is in expectedIPs list , the original-IPs still returned if no ip matched, otherwise act normal and the next-DNS-fallback is performed when no ip matched.

same for unexpectedIPs, if "*" is in unexpectedIPs list, the original-IPs still returned if no ip matched, otherwise act normal and the next-DNS-fallback is performed when no ip matched.

///

in fact, we just replace allowUnexpectedIPs with "*".

@patterniha
Copy link
Contributor Author

patterniha commented May 1, 2025

@RPRX

expectedIPs and unexpectedIPs should behave similarly.

Also, if we have expectedIPs = [ipv4-1, ipv6-1] only ipv4-1 is returned.

In completion of what you said, I will say:

Let's make expectedIPs and unexpectedIPs two separate options.

if "*" is in expectedIPs list , it acts as a priorIPs, otherwise act as expectedIPs.

same for unexpectedIPs, if "*" is in unexpectedIPs list, it acts as a unpriorIPs, otherwise act as unexpectedIPs.

///

in fact, we just replace allowUnexpectedIPs with "*".

I did this.

so if "*" is in expectedIPs or unexpectedIPs, the original-IPs still returned if no ip matched, otherwise they act normal and the next-DNS-fallback is performed when no ip matched.

so new documention is:

(dns-server settings)

disableCache: bool

* The cache is disabled, if disableCache = true in DNS-settings or this DNS-Server-settings.

///

finalQuery:bool

* when true, the result is returned in any case(even when IP-list is empty) and no other fallback will be performed.

///

expectedIPs: [string]

...

if you add "*" in this list, the original-IPs still returned if no ip matched.


///
unexpectedIPs: [string]

* reverse form of expectedIPs, an IP is matched if and only if does not match any of the IP-ranges in the list, in other words: 
`expectedIPs = [0.0.0.0/0, ::/0] - unexpectedIPs.`

if you add "*" in this list, the original-IPs still returned if no ip matched.

///
queryStrategy: "UseIP" | "UseIPv4" | "UseIPv6" | "UseSystem"

when queryStrategy is UseSystem, every time dns-Query call, it check the system-network to see if it supports IPv6(and IPv4) or not, if it support IPv6(or IPv4), the IPv6(or IPv4) is also returned, otherwise not returned.

//////
(dns settings)

useSystemHosts: bool

if true, system-hosts appends to config-hosts at start, default is false.

@patterniha
Copy link
Contributor Author

patterniha commented May 2, 2025

i add new queryStrategy: UseSystem.

built-in-dns should not return IPv6 for the system that not support IPv6 (unless user want to use returned-IP at the other side), this causes several problems that I can explain.

currently for this problem we can create two configs, one for IPv4-network and one for dual-stack-network, but it is not convenient to change the configs all the time.

so i add new queryStrategy: UseSystem,

when queryStrategy is UseSystem, every time dns-Query call, it check the system-network to see if it supports IPv6(and IPv4) or not, if it support IPv6(or IPv4), the IPv6(or IPv4) is also returned, otherwise not returned.

queryStrategy: "UseIP" | "UseIPv4" | "UseIPv6" | "UseSystem"

(we can set at global-dns-settings, or each dns-server-settings)

@patterniha patterniha changed the title DNS: New Features DNS New Features: disableCache, finalQuery, unexpectedIPs, "*", UseSystem May 5, 2025
@j2rong4cn
Copy link
Contributor

i add new queryStrategy: UseSystem.

@patterniha 为什么不用net.InterfaceAddrs()?https://pkg.go.dev/net#InterfaceAddrs

@patterniha
Copy link
Contributor Author

patterniha commented May 5, 2025

i add new queryStrategy: UseSystem.

@patterniha 为什么不用net.InterfaceAddrs()?https://pkg.go.dev/net#InterfaceAddrs

net.dial("udp6", "[2001:4860:4860::8888]:53") doesn't send any data(because it is udp), It just tries to bind a port, If successful, it indicates that the system has an interface that supports IPv6 and routing-table is also set correctly to use that interface.

because [2001:4860:4860::8888] is a real wan-IP, we make sure that routing-table is also ok, and in practice we can bind a port for a wan-IPv6 address.

///

but net.InterfaceAddrs() only show information about interfaces, and it's possible that we can't really bind a port and connect to a wan-IPv6 address.

in addition, net.InterfaceAddrs() also shows link-local and localhost addresses, but we should not consider them.

///

as a result, using udp-dial is definitely a better and more accurate way. (It also costs zero and only binds a port and then release)

@patterniha patterniha mentioned this pull request May 7, 2025
4 tasks
@patterniha
Copy link
Contributor Author

patterniha commented May 7, 2025

in 77027d4 i add useSystemHosts:

if true, system-hosts appends to config-hosts at start: #4677, default is false.

"dns": {
  ...
  "useSystemHosts": true
}

@patterniha patterniha changed the title DNS New Features: disableCache, finalQuery, unexpectedIPs, "*", UseSystem DNS New Features: disableCache, finalQuery, unexpectedIPs, "*", UseSystem-queryStrategy, useSystemHosts May 7, 2025
@patterniha
Copy link
Contributor Author

patterniha commented May 19, 2025

@RPRX

I had a goal (Serverless for Iran) and you helped me a lot to achieve my goal, but you neglected me near the end of my work.

I also helped you and found and fixed nearly 20 small and big bugs in this time.

Serverless-for-iran-anti-sanction-version is not yet complete and needs some-of-these-dns-new-features and happy-eyeballs to complete.

Needless to say, these features are also very useful for Chinese users and other uses.

Many users on Telegram ask me for Serverless-for-iran-anti-sanction-version, According to the tests I did, it opens almost all services and websites (except telegram) in all ISPs in iran.

Why does a PR take more than a month to be approved?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants