Add Docker/k8s sample for Sentinel RLS token server and update document
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
This commit is contained in:
Eric Zhao
parent
73188b46b7
commit
9c268bd89a
|
|
@ -0,0 +1,10 @@
|
||||||
|
FROM java:8-jre
|
||||||
|
|
||||||
|
ENV SENTINEL_HOME /sentinel
|
||||||
|
|
||||||
|
COPY target/sentinel-envoy-rls-token-server.jar $SENTINEL_HOME/
|
||||||
|
|
||||||
|
WORKDIR $SENTINEL_HOME
|
||||||
|
|
||||||
|
ENTRYPOINT ["sh", "-c"]
|
||||||
|
CMD ["java -Dcsp.sentinel.log.dir=/sentinel/logs/ ${JAVA_OPTS} -jar sentinel-envoy-rls-token-server.jar"]
|
||||||
|
|
@ -13,3 +13,47 @@ Build the executable jar:
|
||||||
```bash
|
```bash
|
||||||
mvn clean package -P prod
|
mvn clean package -P prod
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Rule configuration
|
||||||
|
|
||||||
|
Currently Sentinel RLS token server supports dynamic rule configuration via the yaml file.
|
||||||
|
The file may provide rules for one *domain* (defined in Envoy's conf file).
|
||||||
|
In Envoy, one rate limit request might carry multiple *rate limit descriptors*
|
||||||
|
(which will be generated from [Envoy rate limit actions](https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/route/route.proto#envoy-api-msg-route-ratelimit)).
|
||||||
|
One rate limit descriptor may have multiple entries (key-value pair).
|
||||||
|
We may set different threshold for each rate limit descriptors.
|
||||||
|
|
||||||
|
A sample rule configuration file:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
domain: foo
|
||||||
|
descriptors:
|
||||||
|
- resources:
|
||||||
|
- key: "destination_cluster"
|
||||||
|
value: "service_httpbin"
|
||||||
|
count: 1
|
||||||
|
```
|
||||||
|
|
||||||
|
This rule only takes effect for domain `foo`. It will limit the max QPS to 1 for
|
||||||
|
all requests targeted to the `service_httpbin` cluster.
|
||||||
|
|
||||||
|
We need to provide the path to yaml file via the `SENTINEL_RLS_RULE_FILE_PATH` env
|
||||||
|
(or `-Dcsp.sentinel.rls.rule.file` opts). Then as soon as the content in the rule file has been changed,
|
||||||
|
Sentinel will reload the new rules from the file to the `EnvoyRlsRuleManager`.
|
||||||
|
|
||||||
|
We may check the logs in `~/logs/csp/sentinel-record.log.xxx` to see whether the rules has been loaded.
|
||||||
|
We may also retrieve the converted `FlowRule` via the command API `localhost:8719/cluster/server/flowRules`.
|
||||||
|
|
||||||
|
## Configuration items
|
||||||
|
|
||||||
|
The configuration list:
|
||||||
|
|
||||||
|
| Item (env) | Item (JVM property) | Description | Default Value | Required |
|
||||||
|
|--------|--------|--------|--------|--------|
|
||||||
|
| `SENTINEL_RLS_GRPC_PORT` | `csp.sentinel.grpc.server.port` | The RLS gRPC server port | **10240** | false |
|
||||||
|
| `SENTINEL_RLS_RULE_FILE_PATH` | `csp.sentinel.rls.rule.file` | The path of the RLS rule yaml file | - | **true** |
|
||||||
|
| `SENTINEL_RLS_ACCESS_LOG` | - | Whether to enable the access log (`on` for enable) | off | false |
|
||||||
|
|
||||||
|
## Samples
|
||||||
|
|
||||||
|
- [Kubernetes sample](./sample/k8s)
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,105 @@
|
||||||
|
# Sentinel Envoy RLS - Kubernetes sample
|
||||||
|
|
||||||
|
This sample will illustrate how to use Sentinel RLS token server with Envoy in Kubernetes clusters.
|
||||||
|
|
||||||
|
## Build the Docker image
|
||||||
|
|
||||||
|
We could use the pre-built Docker image: `registry.cn-hangzhou.aliyuncs.com/sentinel-docker-repo/sentinel-envoy-rls-server:latest`
|
||||||
|
|
||||||
|
We can also manually build the Docker image in the `sentinel-cluster-server-envoy-rls` directory:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build -t "sentinel/sentinel-envoy-rls-server:latest" -f ./Dockerfile .
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deploy Sentinel RLS token server
|
||||||
|
|
||||||
|
Next we could deploy the Sentinel RLS token server in the K8S cluster.
|
||||||
|
We've provided a deployment template for Sentinel RLS token server in `sentinel-rls.yml`.
|
||||||
|
It includes:
|
||||||
|
|
||||||
|
- A `ConfigMap` that contains the cluster flow control rule for Envoy global rate limiting.
|
||||||
|
This will be mounted as a file in the target `Deployment`, so that the Sentinel RLS token server
|
||||||
|
could load the rules dynamically as soon as the rule in the `ConfigMap` has been updated.
|
||||||
|
- A `Deployment` for Sentinel RLS token server. It will mount the `ConfigMap` as a volume
|
||||||
|
for dynamic rule configuration.
|
||||||
|
- A `Service` that exports the Sentinel command port (8719) and the RLS gRPC port (by default 10245)
|
||||||
|
on a cluster-internal IP so that the Envoy pods could communicate with the RLS server.
|
||||||
|
|
||||||
|
The sample rate limiting rule in the `sentinel-rule-cm`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
domain: foo
|
||||||
|
descriptors:
|
||||||
|
# For requests to the "service_httpbin" cluster, limit the max QPS to 1
|
||||||
|
- resources:
|
||||||
|
- key: "destination_cluster"
|
||||||
|
value: "service_httpbin"
|
||||||
|
count: 1
|
||||||
|
```
|
||||||
|
|
||||||
|
You may enable the access log in the Sentinel RLS token server (output to console)
|
||||||
|
via the `SENTINEL_RLS_ACCESS_LOG` env:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
env:
|
||||||
|
- name: SENTINEL_RLS_ACCESS_LOG
|
||||||
|
value: "on"
|
||||||
|
```
|
||||||
|
|
||||||
|
You may also append JVM opts via the `JAVA_OPTS` env.
|
||||||
|
|
||||||
|
After preparing the yaml template, you may deploy the Sentinel RLS token server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl apply -f sample/k8s/sentinel-rls.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deploy Envoy
|
||||||
|
|
||||||
|
Next we could deploy the Envoy instances in the K8S cluster. If you've already had Envoy instances running,
|
||||||
|
you could configure the address (`sentinel-rls-service`) and the port (`10245`)
|
||||||
|
of the rate limit cluster in your Envoy configuration.
|
||||||
|
|
||||||
|
We've provided a deployment template for Envoy in `envoy.yml`.
|
||||||
|
It includes:
|
||||||
|
|
||||||
|
- A `ConfigMap` that contains the configuration for Envoy.
|
||||||
|
This will be mounted as a file in the target `Deployment`, which will be loaded as the configuration
|
||||||
|
file by Envoy.
|
||||||
|
- A `Deployment` for Envoy. It will mount the `ConfigMap` as a volume
|
||||||
|
for configuration.
|
||||||
|
- A `Service` that exports the Envoy endpoint port (by default 10000) on a cluster-internal IP
|
||||||
|
so that it could be accessible from other pods. If you need external access, you could choose the
|
||||||
|
`LoadBalancer` type or add a frontend ingress.
|
||||||
|
|
||||||
|
In the sample, we have two [Envoy clusters](https://www.envoyproxy.io/docs/envoy/latest/api-v2/clusters/clusters):
|
||||||
|
|
||||||
|
- `service_httpbin`: HTTP proxy to `httpbin.org`
|
||||||
|
- `rate_limit_cluster`: the cluster of the Sentinel RLS token server
|
||||||
|
|
||||||
|
This route configuration tells Envoy to route incoming requests to `httpbin.org`. In the `http_filters` conf item,
|
||||||
|
we added the `envoy.rate_limit` filter to the filter chain so that the global rate limiting is enabled.
|
||||||
|
We set the rate limit domain as `foo`, which matches the item in the Envoy RLS rule.
|
||||||
|
In the `route_config`, we provide the rate limit action: `{destination_cluster: {}}`, which will
|
||||||
|
generate the rate limit descriptor containing the actual target cluster name (e.g. `service_httpbin`).
|
||||||
|
Then we could set rate limit rules for each target clusters.
|
||||||
|
|
||||||
|
After preparing the yaml template, you may deploy the Envoy instance:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl apply -f sample/k8s/envoy.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test the rate limiting
|
||||||
|
|
||||||
|
Now it's show time! We could visit the URL `envoy-service:10000/json` in K8S pods.
|
||||||
|
Since we set the max QPS to 1, we could emit concurrent requests to the URL, and we
|
||||||
|
could see the first request passes, while the latter requests are blocked (status 429):
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Update the rules dynamically
|
||||||
|
|
||||||
|
You could update the rules in the `sentinel-rule-cm` ConfigMap. Once the content is updated,
|
||||||
|
Sentinel will perceive the changes and load the new rules to `EnvoyRlsRuleManager`.
|
||||||
|
|
@ -0,0 +1,132 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: envoy-cm
|
||||||
|
data:
|
||||||
|
envoy-yml: |-
|
||||||
|
admin:
|
||||||
|
access_log_path: /tmp/admin_access.log
|
||||||
|
address:
|
||||||
|
socket_address:
|
||||||
|
protocol: TCP
|
||||||
|
address: 127.0.0.1
|
||||||
|
port_value: 9901
|
||||||
|
static_resources:
|
||||||
|
listeners:
|
||||||
|
- name: listener_0
|
||||||
|
address:
|
||||||
|
socket_address:
|
||||||
|
protocol: TCP
|
||||||
|
address: 0.0.0.0
|
||||||
|
port_value: 10000
|
||||||
|
filter_chains:
|
||||||
|
- filters:
|
||||||
|
- name: envoy.http_connection_manager
|
||||||
|
typed_config:
|
||||||
|
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
|
||||||
|
stat_prefix: ingress_http
|
||||||
|
route_config:
|
||||||
|
name: local_route
|
||||||
|
virtual_hosts:
|
||||||
|
- name: local_service
|
||||||
|
domains: ["*"]
|
||||||
|
routes:
|
||||||
|
- match:
|
||||||
|
prefix: "/"
|
||||||
|
route:
|
||||||
|
host_rewrite: httpbin.org
|
||||||
|
cluster: service_httpbin
|
||||||
|
rate_limits:
|
||||||
|
- stage: 0
|
||||||
|
actions:
|
||||||
|
- {destination_cluster: {}}
|
||||||
|
http_filters:
|
||||||
|
- name: envoy.rate_limit
|
||||||
|
config:
|
||||||
|
domain: foo
|
||||||
|
stage: 0
|
||||||
|
rate_limit_service:
|
||||||
|
grpc_service:
|
||||||
|
envoy_grpc:
|
||||||
|
cluster_name: rate_limit_cluster
|
||||||
|
timeout: 0.25s
|
||||||
|
- name: envoy.router
|
||||||
|
clusters:
|
||||||
|
- name: service_httpbin
|
||||||
|
connect_timeout: 0.5s
|
||||||
|
type: LOGICAL_DNS
|
||||||
|
# Comment out the following line to test on v6 networks
|
||||||
|
dns_lookup_family: V4_ONLY
|
||||||
|
lb_policy: ROUND_ROBIN
|
||||||
|
load_assignment:
|
||||||
|
cluster_name: service_httpbin
|
||||||
|
endpoints:
|
||||||
|
- lb_endpoints:
|
||||||
|
- endpoint:
|
||||||
|
address:
|
||||||
|
socket_address:
|
||||||
|
address: httpbin.org
|
||||||
|
port_value: 80
|
||||||
|
- name: rate_limit_cluster
|
||||||
|
type: LOGICAL_DNS
|
||||||
|
connect_timeout: 0.25s
|
||||||
|
lb_policy: ROUND_ROBIN
|
||||||
|
http2_protocol_options: {}
|
||||||
|
load_assignment:
|
||||||
|
cluster_name: rate_limit_cluster
|
||||||
|
endpoints:
|
||||||
|
- lb_endpoints:
|
||||||
|
- endpoint:
|
||||||
|
address:
|
||||||
|
socket_address:
|
||||||
|
address: sentinel-rls-service
|
||||||
|
port_value: 10245
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1beta2
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: envoy-deployment-basic
|
||||||
|
labels:
|
||||||
|
app: envoy
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: envoy
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: envoy
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: envoy
|
||||||
|
image: envoyproxy/envoy:v1.12.0
|
||||||
|
ports:
|
||||||
|
- containerPort: 10000
|
||||||
|
command: ["envoy"]
|
||||||
|
args: ["-c", "/tmp/envoy/envoy.yaml"]
|
||||||
|
volumeMounts:
|
||||||
|
- name: envoy-config
|
||||||
|
mountPath: /tmp/envoy
|
||||||
|
volumes:
|
||||||
|
- name: envoy-config
|
||||||
|
configMap:
|
||||||
|
name: envoy-cm
|
||||||
|
items:
|
||||||
|
- key: envoy-yml
|
||||||
|
path: envoy.yaml
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: envoy-service
|
||||||
|
labels:
|
||||||
|
name: envoy-service
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
ports:
|
||||||
|
- port: 10000
|
||||||
|
targetPort: 10000
|
||||||
|
protocol: TCP
|
||||||
|
selector:
|
||||||
|
app: envoy
|
||||||
|
|
@ -0,0 +1,68 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: sentinel-rule-cm
|
||||||
|
data:
|
||||||
|
rule-yaml: |-
|
||||||
|
domain: foo
|
||||||
|
descriptors:
|
||||||
|
- resources:
|
||||||
|
- key: "destination_cluster"
|
||||||
|
value: "service_httpbin"
|
||||||
|
count: 1
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1beta2
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: sentinel-rls-server
|
||||||
|
labels:
|
||||||
|
app: sentinel
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: sentinel
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: sentinel
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: sentinelserver
|
||||||
|
# You could replace the image with your own image here
|
||||||
|
image: "registry.cn-hangzhou.aliyuncs.com/sentinel-docker-repo/sentinel-envoy-rls-server:latest"
|
||||||
|
imagePullPolicy: Always
|
||||||
|
ports:
|
||||||
|
- containerPort: 10245
|
||||||
|
- containerPort: 8719
|
||||||
|
volumeMounts:
|
||||||
|
- name: sentinel-rule-config
|
||||||
|
mountPath: /tmp/sentinel
|
||||||
|
env:
|
||||||
|
- name: SENTINEL_RLS_RULE_FILE_PATH
|
||||||
|
value: "/tmp/sentinel/rule.yaml"
|
||||||
|
volumes:
|
||||||
|
- name: sentinel-rule-config
|
||||||
|
configMap:
|
||||||
|
name: sentinel-rule-cm
|
||||||
|
items:
|
||||||
|
- key: rule-yaml
|
||||||
|
path: rule.yaml
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: sentinel-rls-service
|
||||||
|
labels:
|
||||||
|
name: sentinel-rls-service
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
ports:
|
||||||
|
- port: 8719
|
||||||
|
targetPort: 8719
|
||||||
|
name: sentinel-command
|
||||||
|
- port: 10245
|
||||||
|
targetPort: 10245
|
||||||
|
name: sentinel-grpc
|
||||||
|
selector:
|
||||||
|
app: sentinel
|
||||||
Loading…
Reference in New Issue