Yes the colorset matters. I guess I wasn’t clear, FvwmPager uses the foreground color for both the border of the mini window and the label (any text in the mini window) and the background color for the color of the mini window. You cannot have the border and and text be different colors (like the fvwm2 screen shot you had, but that was only a side affect of the windows using the foreground color of active windows for the border of both active and inactive windows).
So what I meant by you cannot get the same thing is I meant you cannot have the border color be black and the text color of the window be white. But you can have the border and text color both be black, in that case do what you did and configure the colorsets to give the foreground and background color you want. If you just want to make the foreground color of the windows black you could just do something like *FvwmPager: Fore black and that would make the border and text black of the inactive windows.