Community detection

In the academic/mathematical literature on community detection, a community within a social network is taken to be a group of users with two characteristics:

• The community is densely connected internally, i.e. people within the same community talk to each other a lot.
• There are relatively few links crossing from the community to the outside world, i.e. people talk to fellow members of their community more often than they talk to non-members.

Community detection is the task of finding communities within a dataset.

We can use community detection to uncover structure in a large digital messaging dataset which gave a way to navigate the large messaging dataset and split it up into meaningful, manageable pieces.

We combined four community detection algorithms:

• the Louvain method, a weighted variant of the Louvain method, the “k-clique communities” method, and second neighbourhoods

We used these algorithms to generate 98,078 candidate communities

• this gave us a wide range of candidates to choose from – all reasonably tight-knit but not just the mathematically best ones.

For each candidate we evaluated a range of statistics

• E.g. size (number of users), average sentiment level, topics of conversation, frequency of members’ participation
• contribution of recently registered users, conductance, a weighted version of conductance.

Using that broader range of statistics, of which conductance was just one, we chose 18 communities of interest.

The image below shows an example of our community detection in action. A standard technique would have placed the two angry-faced people in the top-right with the community in the top-right. Instead, our algorithm puts them in the same community as the other angry-faced individuals.