The function of groups is to help categorize a users's functional roles within your setup. While that sounds fairly similar to departments, there is a distinct difference the 2, not only from semantics but for functional purposes. The main differences between groups and departments are:
a. A group represents a functional purpose. To that reasoning, a user can belong to many groups while only being able to belong to one department.
b. Groups are usually agnostic from an organizational structure viewpoint. Lets say for instance that an organization has 10 departments and each department has a secretary. In this case, Secretary would be a group in itself and the members of the Secretary group would all belong to different departments.
In short, use departments when you are trying to represent a user's placement within and organizational chart and use groups to represent the various functions a user might carry out.
To manage groups, access Setup Users > Setup Groups. Note also that each group created can be tied to a specific organization (but not necessarily so). As mentioned earlier, a user can belong to multiple groups and unless restricted, a group can have members that spans several organizations.