mixing sub sizes?

Prepare to spend money and time.  You should use an active crossover to separate freqs.  And be mindful of the fact that with the sub in the trunk crossed at 60 or 50 Hz, you won't get much out of it.  All the same rules of building proper enclosures would be in effect with the rear-shelf-mounted subs, too.  And the shelf itself will have to be fortified mightily with damping because it will make noise if you don't.

All in all, multiple driver setups are difficult to get good results from.  But if you are willing to put the money and testing and time into it, you can achieve a greater system.  Do you already have the front main speakers separated into 3-ways?  Prob. not....if you did you wouldn't have posted this question, because you will have already experienced what you have to go through to get it right.

The sticky subject Advanced front how-to will give you a lot of info to start with.  Meanwhile, you're much better off replacing the front speakers with a pair of components, since you are obviously not happy with the current tweeter setup.  This will keep your system with just one pair of tweeters in the front soundstage.  Go listen to some different brands.

Per your other question, yes...the 12 and the 8's will be sharing sub duties so you would have to separate sub frequencies accordingly.  You will double or triple your cost of this system, and without benefit of the tuning tools needed (see the linked thread) you are likely to end up with sound that is not any better than keeping it simple.  Consider, instead, a rebuild of the "same old" with new information you can find in this forum.

As S. Kephart said in that thread, "But my question is why?"

Too often this question on mixing sizes comes up not because of strategy, but because those drivers happen to already be on hand.  A careful read of the author's words will indicate where one should go with an answer. 

I'm willing to bet that in most cases there is a dissatisfaction with what gear is currently in place...and that most people don't want to do a complete uninstall of equipment that has at least SOME merit...thus the question of adding to instead of replacing.

People end up with two or three sets of tweeters and mis-matched woofers in their cars....and wonder why they can't get the sound right, so  there should aways be a word of caution.  Stick to the basics unless you've mastered those and have the knowledge and experience to venture into new territories. 

If this thread is based on a strategic endeavor to use two sizes of subs I'll eat my hat.

Myself?  Still trying to master the basics.  And BTW, I have an edible hat tucked away for emergencies...;)

No, multiple subs of the same size should play the same frequencies. Only multiple sized setups will need the varied x-over points.