Fall 2023 Post 05: How to Do Crappy Research

As you already know, I follow a professional blog by Josh Bernoff called Bernoff. Because my students are focused on doing academic research just now, I want to share a couple of posts from Bernoff's blog about research in the real world, the working world, the world that most of you hope to enter after college. Note that Bernoff's posts are pre-LLMs like ChatGPT and Bard, so I will be adding some thoughts about using AI to help you do research in college.

The first post to consider is "The Problem with 'Do Your Own Research'", in which Bernoff explores the mostly sloppy research habits of too many professional people. (By the way, Bernoff was for years an analyst for a large, American research firm, so he made his living by finding and presenting relevant, credible information and insights to help decision-makers make the best decisions they could make. I find his knowledge reliable — most of the time.) You should read his post, but I want to focus on a couple of points.

First, he clarifies the different kinds of research that a professional person can do:

1. Primary research - This research uses strict methodologies to discover new knowledge — as when an epidemiologist uses science to discover how more likely unvaccinated people are to contract Covid than are vaccinated people. Unbiased researchers will report their findings even if the findings contradict the researchers' assumptions or the researchers' funding organizations. Typically, college students don't do this kind of research until they reach graduate school. I certainly didn't.
2. Secondary research - This research uses rigorous methodologies to reveal existing knowledge — as when reporters interview scientists to learn the latest about treating Covid cases. While this kind of research can be very insightful and reliable, these researchers are sometimes biased, reporting only the primary research that supports their point of view and minimizing or ignoring research they don't like. Most undergraduate college students do secondary research, going to the library (in the best case) to learn what the primary researchers say about some issue.
3. Bullshit research - This research uses indiscriminate Google searches or scans Facebook to find out what family and friends and anonymous influencers think about an issue — often just opinion, rumor, conspiracy, and gossip, but no knowledge. This is crappy research. Unfortunately, it's what too many college students and most high school students do. I give those students bad grades to discourage them, but I'm amazed at how they persist.

As I said, most undergraduate students, at best, do secondary research. They collect existing information about an assigned topic and write it up in an academic document of some kind. This is mostly what your undergraduate instructors want you to do: learn something credible about a topic and write it up in a clean document that shows what you learned about nursing or business procedures or computer programming, for instance, and that cites the sources where you got that information. Most of your undergrad professors don't usually expect you to create some new information or new procedure never discovered before (though it does happen). If you do present some new information, then you really better have your primary research evidence at hand to show your professor — otherwise, they might not believe you.

As I also said, too many undergraduates do bullshit research. They do quick Google searches and use the first three items that Google presents, regardless of the source, which they don't pay attention to anyway. What's wrong with this? Bernoff gives us an answer: Google plays to our biases, presenting first the information that it thinks we want to see. Bernoff says, "Google is built to reflect the biases of the searcher." Google's algorithms have spent years analyzing your interests and biases, and they play to those interests in an effort to keep you online longer and on Google's preferred sites more. Google knows your interests and biases better than you do. Want to do research in your nursing class about the Covid pandemic? Google already knows whether or not you take Covid seriously, and their algorithms will present you with the information that plays to your bias. Google's algorithms don't care if the information is reliable so long as it encourages you to click. Facebook is an even worse source of reliable information. And don't believe much of anything you see on television.

I'm not knocking opinion, rumor, and gossip on Facebook or television. They are great sources of information if you want to know what movie to watch or what clothing to buy, but Facebook and Google are not reliable sources for your academic documents. You have to go beyond them. And you've known this intuitively since elementary school. If someone in the third grade told you, "Suzy is your girlfriend," then your first question is "Who says?" In other words you want to know the source of the information so that you can judge how credible and reliable it is. In the third grade, having a girlfriend or boyfriend is a serious charge, and the reliability of the source is critical. You cared about the source, then, because you cared about the issue. Too often in college, you don't care about the issues you're writing about; therefore, you don't care about the sources. Any old quote will do so long as you can get the essay written and off your desk quickly. But remember this: your teachers care about the issues and the sources, and they will give you poor grades if you don't at least fake interest in them as well. Crappy sources angered you in the third grade, and they anger your teachers today. So avoid them. How?

Well, in a second post titled "How to Cite Research that Isn't Crappy", Bernoff explains how to do a good Google search. Unfortunately, he doesn't mention using Google Scholar, but then he isn't writing for an academic audience, either. He definitely doesn't mention using AI such as Bard or ChatGPT because those AI were not available when he wrote the post. Still, he provides four questions that can help determine if a source is worth including in your own document:

1. Who did the research? Is this a source I can trust? Is there a bias?
2. How recent is it? 
3. How many people did they survey? What kinds of people? 
4. What is their sampling method? Is it representative?

Again, keep in mind that he is not writing for an academic audience, but for working professionals. Still, the advice is worth considering for college students. You really must determine up front if you can trust a particular source. Just because something is published on the internet doesn't make it credible. In fact, most sources on the Net are NOT credible. Most web sites exist to influence you to believe something or to buy something. Can you trust someone who is trying to sell you a product, service, or idea to tell you the unbiased truth? Probably not. More importantly, your college instructor won't trust them. So don't cite them in your essays. This is one of the advantages of using sources from the library, GALILEO, or Google Scholar. Most instructors will accept information from those sources.

Definitely don't trust sources listed by a current AI such as Bard or ChatGPT. Not only will Bard send you to poor sources, but it will send you to sources that don't even exist. Bard will make up sources that perfectly fit your topic. Bard aims to please, but it makes up its own targets. I try to get around this bad tendency by asking Bard for ten sources if I only need three. I can usually find three out of the ten that Bard sends me.

Then is your source current? Currency is especially important for business and science oriented essays. Science, technology, and business change too quickly to use old data. How many people and what kind of people did the study you are citing? You have to know who they are and why you can trust them if you expect your instructor to trust them. And if you are using social science sources that use surveys to gather data, then what was their methodology? A sample of six of your friends might be good enough to recommend a new TV show to your mom, but it is hardly substantial enough to warrant a different medical treatment or a change in the laws of a state or the accounting procedures of a company. Fortunately, we in the humanities often rely on and find credible research from older scholarly texts — after all, we are reading stories and poems that are hundreds, even thousands, of years old. Still, literary scholarship in the 19th century is very different from that in the 21st century, so be mindful.

Currency is a particular problem with ChatGPT, which was trained on a dataset that ended at 2021, over two years ago. That is a long time for some fast-moving topics such as business, information technology, and nursing. Because it is connected to the Internet, Bard is aware of more recent work in a range of fields. When I prompted Bard to explain its own access to current data, it responded with:

I am trained on a massive dataset of text and code, including a significant portion of the current world wide web. I can access and process information from the real world through Google Search and keep my response consistent with search results. However, I am still under development, and I cannot access or process all of the information on the web. Additionally, the information that I am trained on is only as up-to-date as the dataset that I was trained on. This means that my responses may not always be the most current.

If you are looking for the most up-to-date information on a particular topic, I recommend that you consult a reputable source such as a news article, scientific journal, or government website. You can also use Google Search to find the most recent information on a topic.

Here are some tips for ensuring that you receive the most accurate and up-to-date information from me:

• Be specific in your queries. The more specific you are, the more likely I am to be able to provide you with accurate and up-to-date information.
• Use relevant keywords. When you are asking me a question, use keywords that are relevant to the topic you are interested in. This will help me to better understand your question and provide you with the most relevant information.
• Be critical of my responses. Just because I provide you with information does not mean that it is accurate. It is important to be critical of my responses and to verify the information that I provide you with.

I think Bard gives sound advice here. Use it.

Remember, in college, you are writing for your instructors, even when they assign another audience as I do. You must gather information that will satisfy the CRAP test (Currency, Reliability, Authority, Purpose) — especially if you want a good grade from your instructor. You must learn to check with your teacher, as different teachers have different measures on the CRAP test: for some like me a Wikipedia article will pass the CRAP test, but for many other teachers, it will not. Be advised.