The two spots you've highlighted appear superficially similar in terms of their geographic setting - that is to say, they are both outcrops of basement (bedrock) that rises out from under (not over) a sea of more recent sediments. Based solely on the aerial photos, I wouldn't be surprised if the abrupt transition from sedimentary basin to uplifted bedrock mountains was a fault (not necessarily recent) in each area, but I haven't looked at a geologic map yet (and the outcrops are, sadly, too distant to visit for a quick inspection from here in Kansas).Here's what I learned. The mountain ranges that pop up out of the desert may themselves be made up of subordinate chunks of rock, called plutons. Each pluton may have formed differently from it's neighbors and hence have different sized mineral crystals. That explains why some spots have nothing but granite and basalt and others just a few miles away may have large chunks of quartz.
You are right to associate crystal size with cooling conditions of the magma. The most direct correlation that geologists generally make is that large crystals are the result of slow cooling of magma, deep underground, whereas smaller crystals are associated with more rapid cooling of magmas that make it to the surface (lavas) or shallow subsurface. It's the rate of cooling rather than the specific cooling temperature that makes the biggest difference.
In detail, it's a function of the degree of undercooling and the nucleation vs. the growth rates of crystals of individual minerals. The presence of magmatic fluids, such as H2O (water, but not in its familiar liquid form) can also aid in the growth of large crystals.
Because individual plutons within the Peninsular Ranges Batholith were intruded at different times and under different conditions, they will have different compositions and textures (crystal sizes).
Our discussion of the melting temperatures of various minerals deserves a simplified example. Rocks are made up of mineral crystals. Each mineral melts at a different temperature. For example, quartz melts at about 1650C. As an aside, the amethysts in the ring you give your sweetheart are varieties of quartz. Mica melts at about 1000C. That means that if a molten deposit of mica and quartz cools slowly enough, the quartz will form large crystals while the mica around it remains liquid. If it cools or crystalizes quickly, then you get a mix of the two and all the rockhound finds is another uninteresting rock.
I believe that our Keeper of the Catacombs is considering a longer post on the subject at his blog. I'm looking forward to it.