Balancing Predefined and User-Generated Topics in Suggestion Systems

1. A user takes a notes on this senior computer science class.
2. When the note is complete and is ready to be published.
3. User enters following topic
4. Defined already in the system
5. computer science,
6. mathematics,
7. data science
8. The class that the user took the note was in a deep learning class
9. He also wants to add the topic
10. deep learning
11. But the problem is deep learning is not pre-defined in the system, so when user types it, shows nothing as suggestion

Current topic suggestion architecture:

Trie reflects only following data:

1. From university service :
2. Majors name
3. includes pretty much all majors in usa
4. computer science, mechanical engineering
5. Universities name
6. includes all universities names in usa
7. University of Louisiana at Lafayette.
8. From user service:
9. Space names
10. includes user create spaces
11. for eg: cmps-101, data-structure, nlp
12. Org names
13. for eg: nsas, unisala
14. Then create a trie data structure.

How is this trie data structure created, when?

Currently this trie can be created using an api.

The important part is: Once the trie is created, it will only reflect all the topics/data that is in the system by the time generation.

For example: we give following data and create a trie.

1. Art
2. Commerce
3. Flim
4. Law
5. Math

now this is our current trie:

A user creates a new space called business. Note that our trie is already generated.

Then that latter space will note be reflected in the topics trie. Since, we created this space after we created our trie, the trie will not reflect the newly added topic of business. And so when user searches business in topics, it will not show up in the search.

In this example, 3 topics i.e computer science, maths, and data science will be sent as payload, whereas the Deep learning is not found as shown in the picture. Thus will not reflected in payload to thread creation api.

**** important ****

Currently new topics cannot be created by adding new topics.

Let's see how our code of trie is:

const { sortWith, descend, prop } = require("ramda");

function TrieNode(value) {
  return {
    char: value,
    isEnd: false,
    keys: {},
    entityType: null, // 'major' or 'school'
    unitId: null,
    pictures: null,
    alias: null,
    universityCount: 0,
    _id: null,
  };
}

class Trie {
  constructor() {
    this.root = {
      root: true,
      keys: {},
    };
  }

  insert(word, entityType, data, node = this.root) {
    if (word.length === 0) {
      node.isEnd = true;
      node.entityType = entityType;
      node.unitId = data.unitId;
      node.pictures = data.pictures;
      node.alias = data.alias;
      node.universityCount = data.universityCount || 0;
      node._id = data._id;
      node.type = data.type;
    } else if (!node.keys[word[0]]) {
      node.keys[word[0]] = TrieNode(word[0]);
      this.insert(word.substring(1), entityType, data, node.keys[word[0]]);
    } else {
      this.insert(word.substring(1), entityType, data, node.keys[word[0]]);
    }
  }

  search(word, node = this.root) {
    if (word.length === 0) {
      return node.isEnd;
    } else if (node.keys[word[0]]) {
      return this.search(word.substring(1), node.keys[word[0]]);
    } else if (!node.keys[word[0]]) {
      node.keys[word[0]] = TrieNode(word[0]);
      return this.search(word.substring(1), node.keys[word[0]]);
    }
  }

  myTree() {
    return this.root;
  }
}

We then insert all the different sort of datas into our trie tree. which will look like this:

For majors:

1. Art
2. Commerce
3. Flim
4. Law
5. Math

Now, these 5 majors are predefined in the trie system.

now if we want to search for business, then we'd not find any suggestion for that. This is the current problem.

How to add anything that is not already predefined.

How can we refine our topic suggestion algorithm to pick user suggested first time topics?

Few things we can brainstorm:

1. Any new first time topics suggested/input by user will be recored
2. will be send new topic as payload
3. We will schedule a trie update at certain time
4. for eg: every day, twice a week or once a week

Challenges:

1. What if user enters random typos
2. aaa, fhsdafljsdakfjas, aksdhjkflsadj, fdsiauiodsaf, ufioasadf
3. anything at all, what happens then?

we will continue this thread discussion in our updated trie system.