Multiple selection list boxes is something I agree needs some work. Unfortunately, without a Ctrl or Shift key, I can't deselect an item when tapping on another item, otherwise you'd never be able to select multiple items.
I have a proposal to help with this. I could make it so multi-select list boxes displayed checkboxes next to each item and used the checked state rather than a selection color to signify which are "selected". That way it's more intuitive that tapping on an item simply toggles the checked state of that item without affecting other items.
What do you think?
EDIT: Just finished implementing this idea. This is what it will look like: