ci: cram: tr181-gre initial tests
Add CRAM tests for tr181-gre, including an end-to-end tunnel test: the remote peer lives in a network namespace behind a veth pair emulating the WAN link, so the test is self-contained and is not tied to a specific board. The end-to-end test drives the tunnel through the data model (GRE.Tunnel + Interface + IP.Interface), verifies the kernel route, reply behaviour with and without a peer return route, the UseSequenceNumber/UseChecksum toggles and runs a short iperf3 session through the tunnel. References: PCF-2294, FEAT-150 Signed-off-by: Khaled Hasni <khaled.hasni_ext@softathome.com> Signed-off-by: Alexandre Fiset <alexandre.fiset@softathome.com>
This commit is contained in:
parent
23e82ac48a
commit
bd85d5aa3e
@ -0,0 +1,96 @@
|
|||||||
|
End-to-end GRE tunnel test (IPv4-over-IPv4). The remote peer runs in a
|
||||||
|
network namespace on the DUT, connected through a veth pair that
|
||||||
|
emulates the WAN link, so the test is self-contained.
|
||||||
|
|
||||||
|
Create R alias:
|
||||||
|
|
||||||
|
$ alias R="${CRAM_REMOTE_COMMAND:-}"
|
||||||
|
|
||||||
|
$ R logger -t cram "Starting GRE E2E test"
|
||||||
|
|
||||||
|
Create the peer network namespace with a veth link emulating the WAN:
|
||||||
|
|
||||||
|
$ R "(ip netns add grepeer; \
|
||||||
|
> ip link add veth-gre type veth peer name veth-grep; \
|
||||||
|
> ip link set veth-grep netns grepeer; \
|
||||||
|
> ip addr add 192.168.78.1/24 dev veth-gre; \
|
||||||
|
> ip link set veth-gre up; \
|
||||||
|
> ip netns exec grepeer ip addr add 192.168.78.2/24 dev veth-grep; \
|
||||||
|
> ip netns exec grepeer ip link set veth-grep up; \
|
||||||
|
> ip netns exec grepeer ip link set lo up) > /dev/null 2>&1"
|
||||||
|
|
||||||
|
Configure the GRE peer endpoint inside the namespace:
|
||||||
|
|
||||||
|
$ R "(ip netns exec grepeer ip tunnel add gre1 mode gre remote 192.168.78.1 local 192.168.78.2; \
|
||||||
|
> ip netns exec grepeer ip addr add 10.0.1.1 dev gre1; \
|
||||||
|
> ip netns exec grepeer ip link set gre1 up) > /dev/null 2>&1"
|
||||||
|
|
||||||
|
Configure the CPE tunnel through the data model:
|
||||||
|
|
||||||
|
$ GRETIDX=$(R "ba-cli 'GRE.Tunnel.+{Alias=\"e2e\",Enable=1,RemoteEndpoints=\"192.168.78.2\",DeliveryHeaderProtocol=\"IPv4\"}'" 2>/dev/null | sed -ne 's/^GRE.Tunnel.\([0-9]\+\).$/\1/p')
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.e2e.Interface.+{Alias=\"i1\",Enable=1}'" > /dev/null 2>&1
|
||||||
|
$ sleep 3
|
||||||
|
$ GREIFACE=$(R "ba-cli 'GRE.Tunnel.e2e.Interface.1.Name?'" | sed -ne 's/.*Name="\(.*\)"$/\1/p')
|
||||||
|
|
||||||
|
Assign an IPv4 address to the CPE GRE interface through the data model:
|
||||||
|
|
||||||
|
$ IPIDX=$(R "ba-cli 'IP.Interface.+{Alias=\"gre-e2e\",Enable=1,IPv4Enable=1,LowerLayers=\"Device.GRE.Tunnel.$GRETIDX.Interface.1.\",Router=\"Device.Routing.Router.1.\"}'" 2>/dev/null | sed -ne 's/^IP.Interface.\([0-9]\+\).$/\1/p')
|
||||||
|
$ R "ba-cli 'IP.Interface.$IPIDX.IPv4Address.+{Alias=\"gre-e2e\",AddressingType=\"Static\",IPAddress=\"10.0.1.2\",SubnetMask=\"255.255.255.0\",Enable=1}'" > /dev/null 2>&1
|
||||||
|
$ sleep 3
|
||||||
|
|
||||||
|
A route to 10.0.1.0/24 through the GRE interface is installed on the CPE:
|
||||||
|
|
||||||
|
$ R "ip route show 10.0.1.0/24 | grep -c \"dev $GREIFACE\""
|
||||||
|
1
|
||||||
|
|
||||||
|
Without a return route on the peer, echo replies do not come back:
|
||||||
|
|
||||||
|
$ R "ping -c 1 -W 2 10.0.1.1 > /dev/null 2>&1; echo \$?"
|
||||||
|
1
|
||||||
|
|
||||||
|
Add the return route on the peer and observe echo replies:
|
||||||
|
|
||||||
|
$ R "ip netns exec grepeer ip route add 10.0.1.2 dev gre1"
|
||||||
|
$ R "ping -c 3 -W 5 10.0.1.1 > /dev/null 2>&1; echo \$?"
|
||||||
|
0
|
||||||
|
|
||||||
|
Enable GRE sequence numbering on the CPE; traffic still passes:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.e2e.Interface.1.UseSequenceNumber=1'" > /dev/null 2>&1
|
||||||
|
$ sleep 2
|
||||||
|
$ R "ip -d link show $GREIFACE | grep -c oseq"
|
||||||
|
1
|
||||||
|
$ R "ping -c 3 -W 5 10.0.1.1 > /dev/null 2>&1; echo \$?"
|
||||||
|
0
|
||||||
|
|
||||||
|
Enable GRE output checksum on the CPE; the peer does not accept
|
||||||
|
checksummed packets yet, so echo replies stop:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.e2e.Interface.1.UseChecksum=1'" > /dev/null 2>&1
|
||||||
|
$ sleep 2
|
||||||
|
$ R "ip -d link show $GREIFACE | grep -c ocsum"
|
||||||
|
1
|
||||||
|
$ R "ping -c 1 -W 2 10.0.1.1 > /dev/null 2>&1; echo \$?"
|
||||||
|
1
|
||||||
|
|
||||||
|
Enable GRE input checksum on the peer; echo replies come back again:
|
||||||
|
|
||||||
|
$ R "ip netns exec grepeer ip link set gre1 type gre icsum"
|
||||||
|
$ R "ping -c 3 -W 5 10.0.1.1 > /dev/null 2>&1; echo \$?"
|
||||||
|
0
|
||||||
|
|
||||||
|
Run a short throughput test through the tunnel:
|
||||||
|
|
||||||
|
$ R "ip netns exec grepeer iperf3 -s -D > /dev/null 2>&1"
|
||||||
|
$ sleep 1
|
||||||
|
$ R "iperf3 -c 10.0.1.1 -t 2 > /dev/null 2>&1; echo \$?"
|
||||||
|
0
|
||||||
|
$ R "killall iperf3 > /dev/null 2>&1; true"
|
||||||
|
|
||||||
|
Cleanup:
|
||||||
|
|
||||||
|
$ R "ba-cli 'IP.Interface.$IPIDX.-'" > /dev/null 2>&1
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.e2e-'" > /dev/null 2>&1
|
||||||
|
$ R "ip netns del grepeer > /dev/null 2>&1; true"
|
||||||
|
|
||||||
|
$ R logger -t cram "GRE E2E test finished"
|
||||||
@ -0,0 +1,79 @@
|
|||||||
|
Create R alias:
|
||||||
|
|
||||||
|
$ alias R="${CRAM_REMOTE_COMMAND:-}"
|
||||||
|
|
||||||
|
Check the GRE datamodel starts empty:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.FilterNumberOfEntries?'" | grep -v '>' | grep -F 'NumberOfEntries='
|
||||||
|
GRE.FilterNumberOfEntries=0
|
||||||
|
$ R "ba-cli 'GRE.TunnelNumberOfEntries?'" | grep -v '>' | grep -F 'NumberOfEntries='
|
||||||
|
GRE.TunnelNumberOfEntries=0
|
||||||
|
|
||||||
|
Make sure we start with no custom GRE devices:
|
||||||
|
|
||||||
|
$ R "ip link show | grep \"gre.*-t.*\" | wc -l"
|
||||||
|
0
|
||||||
|
|
||||||
|
Create a GRE tunnel instance with invalid configuration:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.+{Alias=\"test_tunnel\", Enable=1, DeliveryHeaderProtocol=\"IPv6\", RemoteEndpoints=\"203.0.113.5\"}'" > /dev/null 2>&1
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Status?'" | grep -v '>' | grep -F 'Status='
|
||||||
|
GRE.Tunnel.[0-9]+.Status="Error" (re)
|
||||||
|
|
||||||
|
Ensure no GRE device is created for an invalid tunnel configuration:
|
||||||
|
|
||||||
|
$ R "ip link show | grep \"gre.*-t.*\" | wc -l"
|
||||||
|
0
|
||||||
|
|
||||||
|
Provide valid tunnel configuration:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.DeliveryHeaderProtocol=\"IPv4\"'" > /dev/null 2>&1
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Status?'" | grep -v '>' | grep -F 'Status='
|
||||||
|
GRE.Tunnel.[0-9]+.Status="Enabled" (re)
|
||||||
|
|
||||||
|
Create underlying traffic flow:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Interface.+{Alias=\"test_interface\", Enable=1}'" > /dev/null 2>&1
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Interface.test_interface.Status?'" | grep -v '>' | grep -F 'Status='
|
||||||
|
GRE.Tunnel.[0-9]+.Interface.[0-9]+.Status="Unknown" (re)
|
||||||
|
|
||||||
|
Ensure GRE device is created:
|
||||||
|
|
||||||
|
$ R "ip link show | grep \"gre.*-t.*\" | sed 's/^[0-9]*: //'"
|
||||||
|
gre[0-9]+-t[0-9]+@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 (re)
|
||||||
|
|
||||||
|
Verify NetModel interface status:
|
||||||
|
|
||||||
|
$ R "ba-cli 'NetModel.Intf.[Alias == \"gre-test_interface\"].Status_ext?'" | grep -v '>' | grep -F 'Status_ext='
|
||||||
|
NetModel.Intf.[0-9]+.Status_ext="Unknown" (re)
|
||||||
|
|
||||||
|
Create a second interface:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Interface.+{Alias=\"test_interface2\", Enable=1}'" > /dev/null 2>&1
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Interface.test_interface2.Status?'" | grep -v '>' | grep -F 'Status='
|
||||||
|
GRE.Tunnel.[0-9]+.Interface.[0-9]+.Status="NotPresent" (re)
|
||||||
|
|
||||||
|
Provide a unique key identifier:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Interface.test_interface2.KeyIdentifierGenerationPolicy=\"Provisioned\"'" > /dev/null 2>&1
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Interface.test_interface2.KeyIdentifier=1'" > /dev/null 2>&1
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel.Interface.test_interface2.Status?'" | grep -v '>' | grep -F 'Status='
|
||||||
|
GRE.Tunnel.[0-9]+.Interface.[0-9]+.Status="Unknown" (re)
|
||||||
|
|
||||||
|
Verify NetModel interface status:
|
||||||
|
|
||||||
|
$ R "ba-cli 'NetModel.Intf.[Alias == \"gre-test_interface\"].Status_ext?'" | grep -v '>' | grep -F 'Status_ext='
|
||||||
|
NetModel.Intf.[0-9]+.Status_ext="Unknown" (re)
|
||||||
|
$ R "ba-cli 'NetModel.Intf.[Alias == \"gre-test_interface2\"].Status_ext?'" | grep -v '>' | grep -F 'Status_ext='
|
||||||
|
NetModel.Intf.[0-9]+.Status_ext="Unknown" (re)
|
||||||
|
|
||||||
|
Ensure GRE device is created
|
||||||
|
|
||||||
|
$ R "ip link show | grep \"gre.*-t.*\" | sed 's/^[0-9]*: //'"
|
||||||
|
gre[0-9]+-t[0-9]+@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 (re)
|
||||||
|
gre[0-9]+-t[0-9]+@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1472 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 (re)
|
||||||
|
|
||||||
|
Cleanup:
|
||||||
|
|
||||||
|
$ R "ba-cli 'GRE.Tunnel.test_tunnel-'" >/dev/null 2>&1
|
||||||
@ -1185,6 +1185,67 @@ Device.Firewall.X_PRPLWARE-COM_WANAccess.BlockList.{i}.InterfaceName
|
|||||||
Device.Firewall.X_PRPLWARE-COM_WANAccess.BlockList.{i}.PhysAddress proto::tr181-firewall
|
Device.Firewall.X_PRPLWARE-COM_WANAccess.BlockList.{i}.PhysAddress proto::tr181-firewall
|
||||||
Device.Firewall.X_PRPLWARE-COM_WANAccess.BlockList.{i}.Tags proto::tr181-firewall
|
Device.Firewall.X_PRPLWARE-COM_WANAccess.BlockList.{i}.Tags proto::tr181-firewall
|
||||||
Device.Firewall.X_PRPLWARE-COM_WANAccess.BlockListNumberOfEntries proto::tr181-firewall
|
Device.Firewall.X_PRPLWARE-COM_WANAccess.BlockListNumberOfEntries proto::tr181-firewall
|
||||||
|
Device.GRE. proto::tr181-gre
|
||||||
|
Device.GRE.Filter.{i}. proto::tr181-gre
|
||||||
|
Device.GRE.Filter.{i}.Alias proto::tr181-gre
|
||||||
|
Device.GRE.Filter.{i}.AllInterfaces proto::tr181-gre
|
||||||
|
Device.GRE.Filter.{i}.DSCPMarkPolicy proto::tr181-gre
|
||||||
|
Device.GRE.Filter.{i}.Enable proto::tr181-gre
|
||||||
|
Device.GRE.Filter.{i}.Interface proto::tr181-gre
|
||||||
|
Device.GRE.Filter.{i}.Order proto::tr181-gre
|
||||||
|
Device.GRE.Filter.{i}.Status proto::tr181-gre
|
||||||
|
Device.GRE.FilterNumberOfEntries proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}. proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Alias proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.ConnectedRemoteEndpoint proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.DefaultDSCPMark proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.DeliveryHeaderProtocol proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Enable proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}. proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Alias proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Enable proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.KeyIdentifier proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.KeyIdentifierGenerationPolicy proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.LastChange proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.LowerLayers proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Name proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.ProtocolIdOverride proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats. proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.BroadcastPacketsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.BroadcastPacketsSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.BytesReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.BytesSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.DiscardChecksumReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.DiscardPacketsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.DiscardPacketsSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.DiscardSequenceNumberReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.ErrorsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.ErrorsSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.MulticastPacketsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.MulticastPacketsSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.PacketsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.PacketsSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.Reset() proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.UnicastPacketsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.UnicastPacketsSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Stats.UnknownProtoPacketsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.Status proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.UseChecksum proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Interface.{i}.UseSequenceNumber proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.InterfaceNumberOfEntries proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.KeepAlivePolicy proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.RemoteEndpoints proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats. proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats.BytesReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats.BytesSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats.ErrorsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats.ErrorsSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats.KeepAliveReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats.KeepAliveSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats.PacketsReceived proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Stats.PacketsSent proto::tr181-gre
|
||||||
|
Device.GRE.Tunnel.{i}.Status proto::tr181-gre
|
||||||
|
Device.GRE.TunnelNumberOfEntries proto::tr181-gre
|
||||||
Device.Hardware.
|
Device.Hardware.
|
||||||
Device.Hardware.CPUs. proto::tr181-cpu
|
Device.Hardware.CPUs. proto::tr181-cpu
|
||||||
Device.Hardware.CPUs.CPU.{i}. proto::tr181-cpu
|
Device.Hardware.CPUs.CPU.{i}. proto::tr181-cpu
|
||||||
|
|||||||
@ -92,6 +92,7 @@ Check that ubus has expected datamodels available:
|
|||||||
Device.DynamicDNS
|
Device.DynamicDNS
|
||||||
Device.Ethernet
|
Device.Ethernet
|
||||||
Device.Firewall
|
Device.Firewall
|
||||||
|
Device.GRE
|
||||||
Device.Hardware
|
Device.Hardware
|
||||||
Device.Hardware.CPUs
|
Device.Hardware.CPUs
|
||||||
Device.Hardware.X_PRPLWARE-COM_FlashDevice
|
Device.Hardware.X_PRPLWARE-COM_FlashDevice
|
||||||
@ -186,6 +187,9 @@ Check that ubus has expected datamodels available:
|
|||||||
Firewall.X_PRPLWARE-COM_InterfaceSetting
|
Firewall.X_PRPLWARE-COM_InterfaceSetting
|
||||||
Firewall.X_PRPLWARE-COM_WANAccess
|
Firewall.X_PRPLWARE-COM_WANAccess
|
||||||
Firewall.X_PRPLWARE-COM_WANAccess.BlockList
|
Firewall.X_PRPLWARE-COM_WANAccess.BlockList
|
||||||
|
GRE
|
||||||
|
GRE.Filter
|
||||||
|
GRE.Tunnel
|
||||||
GenericNetworkInterface
|
GenericNetworkInterface
|
||||||
GenericNetworkInterface.Interface
|
GenericNetworkInterface.Interface
|
||||||
Hardware
|
Hardware
|
||||||
|
|||||||
@ -23,6 +23,7 @@ PPW-1729, PPW-1731 and PPW-1786:
|
|||||||
/etc/amx/timingila/extensions/timingila-rlyeh-security/timingila-rlyeh-security-definition.odl
|
/etc/amx/timingila/extensions/timingila-rlyeh-security/timingila-rlyeh-security-definition.odl
|
||||||
/etc/amx/timingila/softwaremodules_definition.odl
|
/etc/amx/timingila/softwaremodules_definition.odl
|
||||||
/etc/amx/tr181-cpu/tr181-cpu_definition.odl
|
/etc/amx/tr181-cpu/tr181-cpu_definition.odl
|
||||||
|
/etc/amx/tr181-gre/tr181-gre_definition.odl
|
||||||
/etc/amx/tr181-usb/tr181-usb_port.odl
|
/etc/amx/tr181-usb/tr181-usb_port.odl
|
||||||
|
|
||||||
$ R "grep -r %usersetting /etc/amx | grep -vE '(upc.odl|.*default.*)' "\
|
$ R "grep -r %usersetting /etc/amx | grep -vE '(upc.odl|.*default.*)' "\
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user