Silicone caulk problems with new replacement vinyl windows

My husband and I are replacing the 1960's aluminum windows in our brick house with new double-pane vinyl windows. We have four installed, eight more to go.

I used standard "acrylic latex plus silicone" white caulk on the window interiors, no problem. On the brick exterior the caulking is tricky because the caulk has to bridge the gap between the vinyl window and the bricks, as well as the mortar gaps between bricks. Lots of meticulous caulk tooling.

The vinyl windows are tan in color, so I chose GE Silicone II 100% Silicone Window and Door Sealant for the exterior, because it was the best color match for the tan windows. Also, I read that silicone was a good quality caulk with good adhesion. It's a bear to apply compared to good old acrylic latex, since it's not water soluble, but I've hung in there. I used foam backer rod in the gaps to provide a good foundation for the caulk.

The problem is, I noticed today that the silicone caulk on the first window we completed (about a month ago) is pulling away from the vinyl window. I don't want to complete the other windows and watch the caulk fail.

First question, has anyone else had a problem with silicone caulk not adhering well to vinyl windows? I thought this stuff was supposed to stick well to any hard surface. It's sticking to the brick and mortar, but not to the smooth surface of the vinyl window edge. Anyone know why?

Second, since I now have to rip out and replace the caulk on the first window (and maybe others) does anyone have a recommendation for a better caulk to use? It has to be something that I can find in a tan color, white will not work for our house exterior.

Our entire window replacement project is now at a standstill until we can resolve this caulk problem. Any help is greatly appreciated...winter is coming and I am getting really frustrated!
Thanks folks!
Sheri