As a thumb rule, avoid Sony and (maybe) Blaupunkt also. I have heard some fishy stories about Blaupunkt audio parts (from employees) and Sony simply doesn't give SQ oriented products as far as car audio is concerned.
You can upgrade in stages (I did in 5 stages!) which will give 2 benefits-
1. You can get better products without the need to upgrade in some years time
2. You can get a better idea about your needs
3. Well planned stages can be better financially too!
To start off, you don't want to change your head unit, so you can follow this plan.
Stage I - Upgrade front and rear speakers. Try out this setup for 1/2 months and find out whats missing in the audio experience.
Stage II - Add an amplifier. You may need to add a Converter for additional outputs if your HU doesn't have (normally stock HUs don't have outputs). Try this setup out with all your speakers powered by the amplifier for 3/4 months and then upgrade if you feel the need to get a subwoofer/bass tube.
Stage III - Since yours is a Vista, damping the doors would be alright as it is prone to rattles and NVH will also improve, along with the sound quality. The outside noises will stay outside.
Then, accordingly, you can plan for more! We are always here to help.
For the speakers, 5.4" size may be hard to find. I have seen a good range of 5.5" but not sure if you'd be able to fit those on the space provided without using spacers. I would suggest 5.5" as they would give better choice of speakers and it is normal to fit spacers nowadays! You can get Hertz components at 5-7k in the aftermarket. Other brands are Pioneer, JBL, etc. but IMO
Hertz would be fine. You will have to take a demo before deciding.
For the rears, you will obviously have to listen to it before buying. So that the product suits your taste and you'll not have to regret afterwards. Preferred brands are Infinity, Pioneer, JBL, etc. with JBL being my choice if you plan to get 6x9 ovals on a tray, which would obviously be better.
Do let me know more about your tastes, preferences in music and other details for better suggestions.
I would advice you to get the products from shops rather than online. I am using RF1400T basstube in my Liva and I am not satisfied with the result. Earlier, I was using JBL GTX1200 subwoofer in a sealed enclosure. The RF1400T is a lot more boomier than the JBL 1200w and even 1300w and has a more deep and growling bassline. It is obviously better than JBL tubes but lacks the punch and in-your-face bass effect. It does sound loud from the outside and is only good for that. I got it for Rs. 5200 with Bill/Warranty at an aftermarket shop in Faridabad, 2 months back. If you try hard in aftermarket shops, you can get better (and very much better) prices than online, plus you can always demo the products.
Wiring would depend upon your budget. For the speaker upgrade, there is no need for additional wiring. If you get them done at an aftermarket shop, then you may get the labour charges waived off, as in most cases. Else, they may be around 1000 bucks maximum for only speakers.
For the amplifier, you will need an Amp wiring kit. Range starts from as low as Rs.300 (in Faridabad) and goes all the way upto Rs. 8000. I am using Fusion wiring in my car and I am okay as there are no issues related to them. Fusion wires costed Rs.1500.
Regarding the point of the products working together, you just need to check what your stock HU can power. The RMS of your speakers should be under the range your HU can provide. If going for amplifier, you will need a 70W RMS Amplifier for powering a 300W RMS subwoofer (just an example) in bridged mode and your speakers should be under 70W RMS as well.
Sorry for a really long reply! Do ask any questions if there.
EDIT: Posting my car's setup if you want to know my experience.
HU: Stock HU given by Toyota
Speakers Front: JBL MS Series Components
Speakers Rear: Empty
Sub: Rockford Fosgate RF1400T Basstube (earlier used a JBL GTX1200 Sub)
Amp: JBL GXA644SI (bought from Snapdeal.com) (earlier used a Sony 2 Channel)
Damping: Dampmat Sheets (for Front Doors and Boot)
Wiring: Fusion Amp Kit
Converter: JBL Converter for creating Pre-outs