They have to be VST2, VST3's are not compatible.
Go to 'File > Plugin Browser" and verify the plugin is found and is enabled. By default they are usually disabled. If it's not found click 'Browse' and locate it (C:\ProgramFiles\VSTPlugins or something similar.)
To add a VST track go to "Score Setup" and click on the right hand list "VST Instruments" it should bring up a list of enabled instruments above it (most of my VST's I use rely on Kontakt as the sampler, so I usually load that up, but yours may very)
To change an instrument to a different VST use the gear to the left of the track in the "Score Setup" and choose from the pull down "VST Instrument" It will then have two columns choose both the left and the right column, if you don't choose the item in the righthand column the instrument won't change.
To quickly bring up the instrument's panel ctrl+i when on the track in score view.
