If you have the type I did with the small, center pin, I found I just kept bending or breaking them when I tried to use a hammer or mallet to drive them in.
The best way I found to insert them was to use a brad driver. Kind of hard to explain without photos (which I have somewhere) but this was the type of brad driver where you inserted the brad in the hole at the end and then give the end of the tool a swift push to drive the pin into the rivet.
Totally confused now? Take a look at this and I think you'll get the idea of how it might work: