Hello there,
Hard to judge without the original recorded tracks since this might be all linked or at least fixable in the mix. When i take listen the quality isn't the first shocking me but more the volume, the peaks of the guit and really to hi sound . And really too much spreaded etc... All of this ending with a poor overall quality indeed.
If you feel it, just send me the original takes i could tell you how i find the recording quality and what you might do to improve it and especially process your audio in order to get a lot more proper result ! ;)
I'm pretty sure it's not related to your specs or software but more to some wrong or forgotten actions (which is normal since we can't all know everything about everything ^^ ).
"Tried the limiter plug-in and compressor to boost the volume but just becomes too distorted and/or muffled. Too much hissing in the track also." - Indeed since those has to be used in some specifics way and oftenly along other processing. Distortion was caused by compression or limiter because you were driving too much volume in them, there is multiple techniques to avoid it, first one being your gain structure ( record at ideal level, mix not too loud leave some headroom , master without overcompressing and erasing the dynamics variations....
Regarding the hissing, you could use some stuff like Izotope RX wich allows to denoise, de crackle, de-clip/click and lot more... But it's expensive you could also find a lot of people offering audio repair services for not so expensive, so dependings on your needs you can choose the better option.
Would be glad to help you with this one if you wan't, denoise, mixing and master (for free), just hit me up.
Hope this might help, kind regards.
