I finally get the keyboard to work fine. As thomas suggested it, it was necessary to use setxkbmap into .xinitrc, and after a lot of search, try and retry, I found the right syntax:
setxkbmap -layout 'ch(fr)' -model pc105 -option terminate:ctrl_alt_bksp,keypad:pointerkeys
Note the single quotes around the layout and the ().