Introduction to methodology

Let’s set up a little bit of notation. We have a population of messages $m_i, i = 1..360,000$. We have selectors $s_k, k = 1..40$. We have a labeled sample $m_{jk}, k= 1..33,000$. Our goal is to estimate the probability for each message that each selector is relevant, $p_{ik}$.

The simplest way to understand the problem is that it is a form of binary classification. We can develop some kind of features or independent variables for each message, then predict each selector with a logistic regression or random forest or similar algorithm. This is in essence what we did, but there are a few interesting quirks.

We were acutely aware that most of the messages are irrelevant to our partners. Only a few messages–about 3000–had one or more selectors identified, and many selectors had only a few relevant messages. The sparseness of the selector relevance means that we needed to be careful not to include too many predictor variables. Therefore a big part of the feature generation is concentrating the useful information into a small number of variables; this is a kind of dimensionality reduction.

Each step of this project, including data prep, feature generation, modeling, and evaluation, was performed using a combination of R and Python, and we used feather for seamless data sharing across tasks and environments.

Features

There’s not quite enough data to do feature discovery, so we had to generate features manually. There are some obvious features: size of the message, time and date information, the presence and type of attachment. Individual messages are bundled into conversations, and we can generate a few more features from the conversation-level: number of messages in the conversation, time period during which the conversation took place, the app used for the conversation (e.g. WhatsApp or Facebook Messenger). However, most of the useful information is in two less-obviously-quantifiable elements: the content of the message, and the people involved in the conversation.

We decided early on that the best way to use message content for selector prediction was to perform topic modeling on the text of the chat. This involves three steps: preparing data; creating document-word and term frequency-inverse document frequency vectors; and modeling the topics.

Data preparation is the easiest. We started with a csv file with two columns, a message id and the content of the chat message. After reading it into a pandas dataframe, we clean it up in python like this:

    texts = list()
    for chat in docs.content:
        chat = unidecode(chat.lower())
        chat = punct_re.sub("", chat)
        text = [word.strip() for word in chat.split()
                if word not in stoplist and _good_w(word)]
        text = [swap.get(word, word) for word in text]
        text = [lemmadict.get(word, word) for word in text]
        texts.append(' '.join(text))

The unidecode function strips accents; the punct_re regular expression strips punctuation; the _good_w() function just tells us if the token is not a number and more than 2 characters; swap and lemmadict standardize word forms (Spanish lemmatization lists can be found here). Stoplist contains uninformative words (Spanish stopwords can be found here). The result is a “bag of words”, a list of unique words in each chat.

Now it’s easy to use python’s scikit-learn package to create the term frequency/inverse document frequency (tfidf) matrix:

    doc_vzr_tfidf = TfidfVectorizer().fit(texts)
    doc_tfidf = doc_vzr_tfidf.transform(texts)

This matrix represents the presence of each word (“term”) in each document with a weight that indicates how rare the word is (rare words are more interesting than common words).

Now we can model “topics,” that is, groups of words that generally occur together. We looked at latent dirichlet allocation (LDA) but we chose non-negative matrix factorization (NMF) because our tests found that the NMF word clusters seemed more coherent to us. Our seat-of-the-pants finding seems to be consistent with others’ comparisons of LDA with NMF for very short documents like chats and tweets (see e.g., Chen et al. 2019).

We spent some time staring at the topics, but we don’t think they have a lot of substantive interpretation. One of the topics seemed to include mostly profane words and words that refer to genitals; another seemed to refer to money and payments; while a third seemed to collect words about movement, time, and location. Our goal is to organize the words into a few topics for predictions, not to interpret the topics, so we tried not to get too caught up in this step.

The second important aspect of messages is who’s talking: who are the particpants in a chat? The participants are in some ways like words in the chats: there are many more participants than we can include as independent variables, so we’d like to group the participants into cliques. Like topics, the cliques don’t really have any substantive interpretation. We’re just looking for a way to reduce the information from the chat participants into a limited number of independent variables. We used the same procedure to group participant cliques that we used to group words: we organized the names and phone numbers into references to unique people, created a message-participant matrix, transformed it to tf-idf, fit an NMF, and predicted the results.

These three sets of variables were our predictors: simple metadata; content in topics; and participants in cliques. Now we’re ready to predict selectors.