You are correct - the driver door trim panel must be removed to get the storage tray out. Lower the window and remove the plug behind the inside door handle to access the screws. Using a trim removal tool, remove the plastic retainers along the side of the door panel. You have to get the tool in the right spot before you pry them out or you can damage the retainers. Lift the panel up and pull it away from the door. After that, remove the four screws that secure the storage panel in place on the door trim panel.