I didn't think to select everything and then deselect the containers; apparently I had it in my head that that would deselect everything in the containers. This worked great.
Selecting the elements in the listing takes about the same time as selecting them one-by-one on-screen, though. I wasn't able to select an item from the listing and then shift-click to select one farther down and in turn get it to select the entire range of items in the listing. The rubber band works great though now that I know I can select outside of the area I need to grab.
Probably not something you'd make a priority but I did notice that I could shift-click inside a container and rubber-band elements without the container itself being selected. This, however, didn't select anything, but it did draw the gray box. Maybe this is something you could consider for a future version. It just seems more intuitive to me to select the items I actually want to work with.