The function of Groups is to help categorize a users's working roles within your setup. While this sounds fairly similar to departments, there is a distinct difference between the two, not only in semantics but in functional purposes as well. The main differences between groups and departments are:
a. A group represents a functional purpose. Thus, a user can belong to many groups but can only belong to one department.
b. Groups are usually agnostic from an organizational structure viewpoint. Lets say, for instance, that an organization has 10 departments and each department has a secretary. In this case, Secretary would be a group in itself and the members of the Secretary group would all belong to different departments.
In short, use departments when you are trying to represent a user's placement within an organizational chart, and use groups to represent the various functions a user might carry out.
Figure 1: Setup Users - Setup Groups
To manage groups, go to Setup Users > Setup Groups. Note that each group created can be tied to a specific organization, although not necessarily so. As mentioned earlier, a user can belong to multiple groups and, unless restricted, a group can have members that span several organizations.