CourseRank

CourseRank ranks university courses based on their relevance to a user's search query using the Okapi BM25 algorithm. It processes pre-computed topic counts from course descriptions to determine the most relevant courses for a given query.

This project was developed using data scraped by David Tejuosho.

How it Works

The core of CourseRank is the Okapi BM25 algorithm, a standard ranking function used in information retrieval. It scores each course (document) based on:

1. Term Frequency (TF): How often query terms appear in the course description's topics.
2. Inverse Document Frequency (IDF): How common or rare the query terms are across all course descriptions in the dataset. More weight is given to rarer terms.
3. Course Description Length: The algorithm incorporates the length of the course's topic set relative to the average length across all courses.

The final relevance score for a course is calculated by summing the BM25 scores for each term in the user's query.

Implementation

• Language: Python
• Core Libraries: Pandas, NumPy
• Format: The ranking logic is implemented primarily within a Jupyter Notebook (.ipynb).

Data Requirements

CourseRank expects pre-processed course data, primarily:

• Data/courseTopicSets.json: A JSON array where each object contains:

  • courseCode (string): The unique identifier for the course (e.g., "CPSC 220").
  • topicSet (object): A dictionary mapping processed words (topics) from the course description to their frequency count (e.g., {"computer": 3, "science": 2, "data": 1}).

• (Optional): The notebook might also utilize a separate JSON file containing full course details (name, description, etc.) for displaying results, but this is not essential for the ranking calculation itself.

Usage

1. Prerequisites:
   • Python 3
   • Jupyter Notebook or JupyterLab
   • Required libraries: pip install pandas numpy notebook
2. Data:
   • Place your courseTopicSets.json file inside a Data/ directory within the project folder.
3. Run:
   • Open and execute the cells in the Jupyter Notebook (.ipynb file).
4. Query:
   • Locate the section in the notebook where the BM25 function is called or where the search query variable is defined.
   • Modify the query string (e.g., "computer science") to your desired search terms.
5. Output:
   • The notebook will calculate BM25 scores for all courses based on the query and output a ranked list of relevant course codes.

Example Output

For a query like "computer science", the output might look like: Rank 1). CPSC 220 Rank 2). CPSC 438 Rank 3). MATH 170 Rank 4). CPSC 471

Acknowledgements

• Course data originally scraped and provided by David Tejuosho.
• BM25 implementation inspired by resources like Badri Adhikari's explanation and the AUEB overview.