Super Quick: RAG Comparison between GPT4 and Open-Source

Ashhadul Islam

Follow

Published in

Python in Plain English

5 min readNov 14, 2023

--

Who reads PDF files better?

Just an image, nothing symbolic. Midjourney prompt: “**the eternal tussle between david and goliath”**

As we noted in the last OpenAI dev day (more like RAGgers doomsday, [kidding]) GPT4 now has the ability to glean information from pdf files (that you can just drag and drop) and answer questions about it. I wanted to check how well it performs. I also wanted to compare how open source options fare against it.

The open source option

We are going to use the same configuration I had worked on in my last article. Details are here. Basically I create a folder in which I put my research papers. Then I create embeddings using GPT4AllEmbeddings and save a local copy using Chroma. All the code for that is here. Then I use ollama and load a zephyr 7b parameters model. Consequently I run a chainlit interface to interact with the model.

KNNOR is an algorithm mentioned in one of my research papers

Let me put the text in for better readability.

User: What is KNNOR?

Chatbot: KNNOR, short for K Nearest Neighbor OverSampling, is a method used to create new data points in the given dataset. Its practicality lies in three main activities: identifying critical data that will be utilized to generate new points. This approach is helpful in situations where there is a lack of data or imbalanced distribution of data, which leads to poor generalization capabilities in machine learning models. KNNOR’s pre-augmentation step involves calculating the optimized distance between existing data points, followed by creating new data points based on these identified critical ones. While this method results in a longer processing time compared to other oversampling methods like SMOTE, it has been found to be more effective in certain scenarios, particularly when dealing with abstract art data, where each image of a painting has distinct features that make the model prone to overfitting.

Not bad. The information is correct. It gives some pros and cons as well. Lets take a look under the hood to see what is the source of this infomation.

So the vector search used the file Fast_and_Efficient_Image_Generation_Using_Variational_Autoencoders_and_K-Nearest_Neighbor_OveRsampling_Approach.pdf to get the answer.

It is important to know the source document because I am going to upload the same pdf into GPT4 and ask the same question.

GPT4 has entered the building

I need to buy a subscription first.

Once you buy a subscription with 20USD per month, you get the option to switch between GPT3.5 and 4.

Next we test GPT4 by asking the same question without giving the pdf file.

Utter lies. I just could not find anything called **K-Nearest Neighbors for Ordinal Regression.** GPT4 just made it up

So when I ask what is KNNOR, GPT4 gives a made up full-form and then proceeds to explain K Nearest Neighbors algorithm (which is just KNN) then it gives more made up stuff about KNN for Ordinal Regression (KNNOR). I searched high and low but could not find anything. This is expected I believe.

Moment of truth

Next i dragged and dropped the same pdf file (Fast_and_Efficient_Image_Generation_Using_Variational_Autoencoders_and_K-Nearest_Neighbor_OveRsampling_Approach.pdf) into GPT4’s chat window.

Once I write the question and press enter, it shows a nice little animation that it is reading the file.

Finally I get the answer.

10/10 for accuracy. It also has links leading to the PDF file. This is useful in cases where we upload multiple pdf file and it references which pdf it got the specific info from.

So GPT4 gets it absolutely right. It gives the overview, the steps in the algorithm, the advantages and the applications. The answer is longer and better organized than ollama (Zephyr). The only edge that Zephyr had is that it had mentioned a limitation of the algorithm, whereas GPT4 only sings praises.