Those little buggers are called Groove Pins. They pretty much need to be hammered out... but you cannot because you're on the wrong side...
I also found that just pulling on them with vice grips does not work so well. This is what I came up with to get them out, and it works very well - A slide hammer attached to vice grips... (I made an adapter to attach a slide hammer to my Vise Grips). Get a good grip on the pin and one or two firm whacks gets them out quickly and easily.

For what it's worth, I had this exact question a few years ago - My whole post on this:
Major TL-15 Groove Pin removal. But, I am now using Super Lube PTFE instead of white lithium as recommended by Bill. It is definitely better.
You do want to get those bolts out and clean up the holes and the underside of the cam plate.