Docker routing into ipvlan from host network - docker

Question

Goal:
I like to be able to ping and access the docker clients from my host network. And if possible, I like to have as much as possible configured in my docker-compose.yml.
Remark:
ICMP (ping) is just used for simplification. Actually, I like to access ssh on 22 and some other ports. Mapping ports is my current solution, but since I have many docker client container it becomes messy.
___________ ___________ ___________
| host | | docker | | docker |
| client | | host | | client |
| ..16.50 | <--> | ..16.10 | | |
| | | ..20.1 | <--> | ..20.5 |
| | | |
| | <----- not working ----> | |
Problem:
I am able to ping my docker host from docker clients and host clients, but not the docker clients from host clients.
That's my configuration on ubuntu 22.04.
docker host: 192.168.16.10/24
client host network: 192.168.16.50/24
default gw host network: 192.168.161 /24
docker client (container): 192.168.20.5 /24
docker-compose.yml
version: '3'
networks:
ipvlan20:
name: ipvlan20
driver: ipvlan
driver_opts:
parent: enp3s0.20
com.docker.network.bridge.name: br-ipvlan20
ipvlan-mode: l3
ipam:
config:
- subnet: "192.168.20.0/24"
gateway: "192.168.20.1"
services:
portainer:
image: alpine
hostname: ipvlan20
container_name: ipvlan20
restart: always
command: ["sleep","infinity"]
dns: 192.168.16.1
networks:
ipvlan20:
ipv4_address: 192.168.20.5
On my docker host, I added the following link with the vlan gateway IP.
ip link add myipvlan20 link enp3s0.20 type ipvlan mode l3
ip addr add 192.168.20.1/24 dev myipvlan20
ip link set myipvlan20 up
And on my host client, I added a rout to the docker host for the docker client network.
ip route add 192.168.20.0/24 via 192.168.16.10
I tried also:
Do I have to use macvlan? I tried that, but also unsuccessfully.
Do I have to use l3? I also tried with l2, but unsuccessfully as well.