You don’t mention the task type when you perform these two options? When adding a resource to a task, Microsoft Project looks at the task type and also whether is a resource already assigned, as well as the effort driven flag and performs the appropriate calculation based on these settings for the task. So I know this will sound a little lame but it will depend on the result you are trying to achieve when working with resources.
I agree with John that based on the task type field the behavior changes.
By default, the task type is set to Fixed Unit (you can open the task and see the Advanced tab, you will notice the Task Type is Fixed Units). In your case you have added work for a task and then assigned the resource. Now being a fixed units task, the duration will be adjusted based on the work and the resource units. If you remove the resource from the task, in other words when you decrease the resource units the work is reduced to fill the task duration and hence the work is reduced to 0.
If you want the task work to be fixed and the duration to be flexible, change the Task Type to Fixed Work. If you are assigning multiple resources to the task and want task work to be distributed across the task duration such that the hours assigned each resource per day is less then set Task Type to Fixed Duration and check Effort Driven (in Advanced tab) – that is Task Work is also fixed