Connect your booth outputs to the inputs on the AudioBox USB.
See callout 16 on page 5 of your DJM-800 User Manual where it says:
16. BOOTH monitor output connectors 6.3 mm phone-type booth monitor output connectors. The sound level from these connectors is controlled independently by the BOOTH MONITOR level dial, regardless of the position of the MASTER LEVEL dial. (These connectors are TRS output, so they support both balanced and unbalanced outputs.)
Run two cables, each being a 1/4" TRS to 1/4"TRS cable from your DJM-800 Booth output to the input on the AudioBox USB. Make sure the input knobs on the AudioBox USB are turned all the way hard left which indicates -10db on the silk screen. Then set the Booth Monitor knob on the DJM-800 to hard left which is minus infinity and slowly bring up the Booth Monitor Knob until you have a level in your recording software so the meters are not peaking red.
Then run the AudioBox USB output to your Booth Monitor Speaker. The AudioBox USB will act as a vampire tap into the audio signal coming from the DJM-800.
You don't want to use the RCA outputs because these are a different impedance meant for a Tape Deck which is not the impedance of a 1/4" Instrument Input on the Audio Interface. You also don't want to use the amplified XLR Outputs because these are going to your Main Amps, if you overdrive the Preamp input on the AudioBox USB you can damage the preamp circuits.
Even though the Booth Monitor output of the DJM-800 is a trim control, you don't want to end up with a preamp driving another preamp, this double attenuation will cause distortion and may harm the preamp circuits on the AudioBox USB. It is imperative that you leave the AudioBox 1/2 inputs set to -10 when setup in this configuration.
For anyone else reading this, the same applies to any audio interface with 1/4" Inputs.