I run the kserve example for sklearn-iris and got `302 Found` - kubeflow

Question

serving exam model
create namespace
$ kubectl create namespace kserve-test
create InferenceService
$ kubectl apply -n kserve-test -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "sklearn-iris"
spec:
predictor:
model:
modelFormat:
name: sklearn
storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
EOF
check
$ kubectl get inferenceservices sklearn-iris -n kserve-test
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
sklearn-iris http://sklearn-iris.kserve-test.example.com True 100 sklearn-iris-predictor-default-00001 5h11m
check SERVICE_HOSTNAME, INGRESS_PORT, INGRESS_HOST
SERVICE_HOSTNAME
$ SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kserve-test -o jsonpath='{.status.url}' | cut -d "/" -f 3)
$ echo $SERVICE_HOSTNAME
sklearn-iris.kserve-test.example.com
INGRESS_PORT
$ INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(#.name=="http2")].nodePort}')
$ echo $INGRESS_PORT
31018
INGRESS_HOST
$ INGRESS_HOST=192.168.219.100
192.168.219.100 : Internal IP of my device
create input
$ cat <<EOF > "./iris-input.json"
{
"instances": [
[6.8, 2.8, 4.8, 1.4],
[6.0, 3.4, 4.5, 1.6]
]
}
EOF
send request
$ curl -v -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d #./iris-input.json
* Trying 192.168.219.100:31018...
* Connected to 192.168.219.100 (192.168.219.100) port 31018 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kserve-test.example.com
> User-Agent: curl/7.71.1
> Accept: */*
> Content-Length: 76
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY3MTU5MDMxOHxFd3dBRUhZek16WktlRlZJZFc1alowVlROVTA9fFynQ-3082qPF_-qUwnYllySrEQPAKGqpBuF-Pu9gcnx
< date: Wed, 21 Dec 2022 02:38:38 GMT
< x-envoy-upstream-service-time: 6
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host 192.168.219.100 left intact
Got 302 Found.
As far as I know, this result is because I was asked for authentication from Dex.
what i did to solve this
Authentication
I tried to get the authservice_session token by following the method here: kserve:github
From CLI
$ CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')
CLUSTER_IP: 10.103.239.220
$ curl -v http://${CLUSTER_IP}
* Trying 10.103.239.220:80...
* Connected to 10.103.239.220 (10.103.239.220) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.103.239.220
> User-Agent: curl/7.71.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< content-type: text/html; charset=utf-8
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY3MTU5MzE2MXxFd3dBRUdGM2JUSnlaMkZVYjFWUVNFa3dURGs9fEDuO8ql3cFsetSfKntLvFV0al5tEZJeh23VK-JrJubM
< date: Wed, 21 Dec 2022 03:26:01 GMT
< content-length: 269
< x-envoy-upstream-service-time: 4
< server: istio-envoy
<
Found.
* Connection #0 to host 10.103.239.220 left intact
I stuck at this stage. I think it's wrong to see 302 Found.
From the browser
Copy the token content from the cookie authservice_session
$ SESSION =MTY3MTUyODQ2M3xOd3dBTkRkRVExbEdVa0kzVFRJMFMwOU9VRE5hV2pSS1VGVkNSRVJVUlRKVlVVOUlTa2hDVWpOU1RUZFRVRkJGVTFGV1N6UktXVkU9fCoQdbMu_diLBJAKLZSmF4qoqQTlINKq7A63hy-QNQcR
$ curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Cookie: authservice_session=${SESSION}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d ./iris-input.json
got 404 Not Found
curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Cookie: authservice_session=${SESSION}" http://${CLUSTER_IP}/v1/models/${MODEL_NAME}:predict -d #./iris-input.json
got 404 Not Found
added Envoy filter for bypass Dex
$ vi envoyfilter.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: sklearn-iris-filter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: VIRTUAL_HOST
match:
routeConfiguration:
vhost:
name: sklearn-iris.kserve-test.example.com:31018
patch:
operation: MERGE
value:
per_filter_config:
envoy.ext_authz:
disabled: true
$ kubectl apply -f envoyfilter.yaml
Note: spec.configPatches.match.routeConfiguration.vhost.name : sklearn-iris.kserve-test.example.com:31018
It's not working.
Got still 302 Found
External Authorization
reference: https://github.com/kubeflow/kubeflow/issues/4549#issuecomment-932259673
add AuthorizationPolicy
$ vi authorizationpolicy.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: dex-auth
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
action: CUSTOM
provider:
# The provider name must match the extension provider defined in the mesh config.
name: dex-auth-provider
rules:
# The rules specify when to trigger the external authorizer.
- to:
- operation:
notPaths: ["/v1*"]
$ kubectl apply -f authorizationpolicy.yaml
Note: rules.to.operation: notPaths: ["/v1*"]
and delete envoyfilters named authn-filter that originally existed
$ kubectl delete -n istio-system envoyfilters.networking.istio.io authn-filter
next, restart deployment/istiod
$ kubectl rollout restart deployment/istiod -n istio-system
It's not working.
Got still 302 Found if don't delete envoyfilters named authn-filter that originally existed, or block the connection if I delete authn-filter .
What I need help:
How can I get Dex authorization and make a connected?
Or how can I bypass Dex if I can't get Dex authorization?
Maybe my model serving example is wrong. Thanks for any advice on what's wrong.
env:
ubuntu 20.04
$ kubectl version --client && kubeadm version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:"a43c0904d0de10f92aa3956c74489c45e6453d6e", GitTreeState:"clean", BuildDate:"2022-08-17T18:28:56Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
kubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:
I installed istio and kanative that included in the kubeflow manifest by following: kubeflow/manifasts.
$ kubectl get pod -n istio-system
istio-system authservice-0 1/1 Running
istio-system cluster-local-gateway-5449f87d9b-bb4vs 1/1 Running
istio-system istio-ingressgateway-77b9d69b74-xmv98 1/1 Running
istio-system istiod-67fcb675b5-kzfvd 1/1 Running
$ kubectl get pod -n knative-eventing
$ kubectl get pod -n knative-serving
knative-eventing eventing-controller-8457bd9747-855lc 1/1 Running
knative-eventing eventing-webhook-69986cfb5d-hn7tx 1/1 Running
knative-serving activator-7c5cd78566-pz6ns 2/2 Running
knative-serving autoscaler-98487645d-vh5wk 2/2 Running
knative-serving controller-7546f544b7-mng9g 2/2 Running
knative-serving domain-mapping-5d56bfc7d-5cb9l 2/2 Running
knative-serving domainmapping-webhook-696559d49c-p8rwr 2/2 Running
knative-serving net-istio-controller-c4d469c-lt5fl 2/2 Running
knative-serving net-istio-webhook-855bcb6747-wbl4x 2/2 Running
knative-serving webhook-59f9fdd446-xsf6n 2/2 Running
And installed KServe and KServe Built-in ClusterServingRuntimes by following: kserve installation
$ kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.9.0/kserve.yaml
$ kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.9.0/kserve-runtimes.yaml
$ kubectl get pod -n kserve
kserve-controller-manager-5fc887875d-m7rlp 2/2 Running
check gateway selector
knative-local-gateway in namespace knative-serving
$ kubectl get gateways knative-local-gateway -n knative-serving -o yaml
spec:
selector:
app: cluster-local-gateway
istio: cluster-local-gateway
istio-ingressgateway in namespace istio-system
$ kubectl get gateways istio-ingressgateway -n istio-system -o yaml
spec:
selector:
app: istio-ingressgateway
istio: ingressgateway
cluster-local-gateway in namespace istio-system
$ kubectl get gateways cluster-local-gateway -n istio-system -o yaml
spec:
selector:
app: cluster-local-gateway
istio: cluster-local-gateway
kubeflow-gateway in namespace kubeflow
$ kubectl get gateways kubeflow-gateway -n kubeflow -o yaml
spec:
selector:
istio: ingressgateway