Move forward with your plan and don't look back.
Having had multiple SLs, Jags, Porsches, Vettes, etc for over 40 years, I can tell you that the XLR is the best vehicle "out of the box" that I've ever had the pleasure of driving.
I've had zero problems, and the XLR is a much more pleasant driving experience than any of the other vehicles out there, inclusive of the new SL.
While the SC is a nicely crafted car, it has no soul, cannot handle with anywhere near the surgical efficiency of the XLR, and has major problems with its combination of run-flat tires and suspension. The two factors do not work in combination on the SC, leaving one with a continually jarring ride.
The XLR handles similarly to a Corvette, but with the comfort of a Cadillac or Lincoln. Its suspension is compliant and well-mannered, while allowing one to straighten the curves in the road. The build quality as well as that of the materials used is superb.
You'll find that the XLR is superb driver's vehicle which, albeit with a small load of luggage with the top down, will give you absolute comfort on a long drive.
On a final point, the SCs are so plentiful (and owners either don't seem to keep them or the majority are leased) that they do not hold up for resale and plummet value-wise. The SLs, which were bringing ridiculous numbers upon release, are now plentiful, with used examples to be found on dealers lots, which have dropped 15K or more as used vehicles.
I hope this is helpful.