As you already know, I follow a professional blog by Josh Bernoff called Josh Bernoff. Because my students are focused on doing academic research just now, I want to share a couple of posts from Bernoff's blog about research in the real world, the working world, the world that most of you hope to enter after college. Note that Bernoff's posts are pre-LLMs like ChatGPT and Gemini, so I will be adding some thoughts about using AI to help you do research in college.
The first post to consider is "The Problem with 'Do Your Own Research'", in which Bernoff explores the mostly sloppy research habits of too many professional people. (By the way, Bernoff was for years an analyst for a large, American research firm, so he made his living by finding and presenting relevant, credible information and insights to help decision-makers make the best decisions they could make. I find his knowledge reliable — most of the time.) You should read his post, but I want to focus on a couple of points.
First, he clarifies the different kinds of research that a professional person can do:
- Primary research - This research uses strict methodologies to discover new knowledge — as when an epidemiologist uses science to discover how more likely unvaccinated people are to contract Covid than are vaccinated people. Unbiased researchers will report their findings even if the findings contradict the researchers' assumptions or the researchers' funding organizations. Typically, college students don't do this kind of research until they reach graduate school. I certainly didn't.
- Secondary research - This research uses rigorous methodologies to reveal existing knowledge — as when reporters interview scientists to learn the latest about treating Covid cases. While this kind of research can be very insightful and reliable, these researchers are sometimes biased, reporting only the primary research that supports their point of view and minimizing or ignoring research they don't like. Most undergraduate college students do secondary research, going to the library (in the best case) to learn what the primary researchers say about some issue.
- Bullshit research - This research uses indiscriminate Google searches or scans Facebook to find out what family and friends and anonymous influencers think about an issue — often just opinion, rumor, conspiracy, and gossip, but no knowledge. This is crappy research. Unfortunately, it's what too many college students and most high school students do. I give those students bad grades to discourage them, but I'm amazed at how they persist.
As I said, most undergraduate students, at best, do secondary research. They collect existing information about an assigned topic and write it up in an academic document of some kind. This is mostly what your undergraduate instructors want you to do: learn something credible about a topic and write it up in a clean document that shows what you learned about nursing or business procedures or computer programming, for instance, and that cites the sources where you got that information. Most of your undergrad professors don't usually expect you to create some new information or new procedure never discovered before (though it does happen). If you do present some new information, then you really better have your primary research evidence at hand to show your professor — otherwise, they might not believe you.
As I also said, too many undergraduates do bullshit research. They do quick Google searches and use the first three items that Google presents, regardless of the source, which they don't pay attention to anyway. What's wrong with this? Bernoff gives us an answer: Google plays to our biases, presenting first the information that it thinks we want to see. Bernoff says, "Google is built to reflect the biases of the searcher." Google's algorithms have spent years analyzing your interests and biases, and they play to those interests in an effort to keep you online longer and on Google's preferred sites more. Google knows your interests and biases better than you do. Want to do research in your nursing class about the Covid pandemic? Google already knows whether or not you take Covid seriously, and their algorithms will present you with the information that plays to your bias. Google's algorithms don't care if the information is reliable so long as it encourages you to click. Facebook is an even worse source of reliable information. And don't believe much of anything you see on television.I'm not knocking opinion, rumor, and gossip on Facebook or television. They are great sources of information if you want to know what movie to watch or what clothing to buy, but Facebook and Google are not reliable sources for your academic documents. You have to go beyond them. And you've known this intuitively since elementary school. If someone in the third grade told you, "Suzy is your girlfriend," then your first question is "Who says?" In other words you want to know the source of the information so that you can judge how credible and reliable it is. In the third grade, having a girlfriend or boyfriend is a serious charge, and the reliability of the source is critical. You cared about the source, then, because you cared about the issue. Too often in college, you don't care about the issues you're writing about; therefore, you don't care about the sources. Any old quote will do so long as you can get the essay written and off your desk quickly. But remember this: your teachers care about the issues and the sources, and they will give you poor grades if you don't at least fake interest in them as well. Crappy sources angered you in the third grade, and they anger your teachers today. So avoid them. How?
Well, in a second post titled "How to Cite Research that Isn't Crappy", Bernoff explains how to do a good Google search. Unfortunately, he doesn't mention using Google Scholar, but then he isn't writing for an academic audience, either. He definitely doesn't mention using AI such as Gemini or ChatGPT because those AI were not available when he wrote the post. Still, he provides four questions that can help determine if a source is worth including in your own document:
- Who did the research? Is this a source I can trust? Is there a bias?
- How recent is it?
- How many people did they survey? What kinds of people?
- What is their sampling method? Is it representative?
Again, keep in mind that he is not writing for an academic audience, but for working professionals. Still, the advice is worth considering for college students. You really must determine up front if you can trust a particular source. Just because something is published on the internet doesn't make it credible. In fact, most sources on the Net are NOT credible. Most web sites exist to influence you to believe something or to buy something. Can you trust someone who is trying to sell you a product, service, or idea to tell you the unbiased truth? Probably not. More importantly, your college instructor won't trust them. So don't cite them in your essays. This is one of the advantages of using sources from the library, GALILEO, or Google Scholar. Most instructors will accept information from those sources.
Definitely don't trust sources listed by a current AI such as Gemini or ChatGPT. Not only will Gemini send you to poor sources, but it will send you to sources that don't even exist. Gemini will make up sources that perfectly fit your topic. Gemini aims to please, but it makes up its own targets. I try to get around this bad tendency by asking Gemini for ten sources if I only need three. I can usually find three out of the ten that Gemini sends me.
Then is your source current? Currency is especially important for business and science oriented essays. Science, technology, and business change too quickly to use old data. How many people and what kind of people did the study you are citing? You have to know who they are and why you can trust them if you expect your instructor to trust them. And if you are using social science sources that use surveys to gather data, then what was their methodology? A sample of six of your friends might be good enough to recommend a new TV show to your mom, but it is hardly substantial enough to warrant a different medical treatment or a change in the laws of a state or the accounting procedures of a company. Fortunately, we in the humanities often rely on and find credible research from older scholarly texts — after all, we are reading stories and poems that are hundreds, even thousands, of years old. Still, literary scholarship in the 19th century is very different from that in the 21st century, so be mindful.
Currency is a particular problem with ChatGPT, which was trained on a dataset that ended at 2021, over two years ago. That is a long time for some fast-moving topics such as business, information technology, and nursing. Because it is connected to the Internet, Gemini is aware of more recent work in a range of fields. When I prompted Gemini to explain its own access to current data, it responded with:
Gemini is a large language model developed by Google AI. It was trained on a massive dataset of text and code, allowing it to generate human-quality text, translate languages, write different kinds of creative content, and answer your questions in an informative way.
One of Gemini's strengths is its ability to access and process real-world information, making it a valuable tool for academic research. t can provide you with information on a wide range of topics, from current events to historical facts. When you're researching for your academic papers, Gemini can help you find relevant sources and connect you to the latest information. For example, if you're writing a paper on climate change, Gemini can help you find recent studies, news articles, and government reports on the topic.
However, it's important to note that Gemini's understanding of the world is limited by the data it was trained on. While it can provide you with information on a wide range of topics, it may not always have the most up-to-date or comprehensive data. For example, if a new scientific discovery is made just after Gemini's training data is finalized, it may not be aware of it.
To overcome these limitations, students can supplement their use of Gemini with additional research. This might involve searching for articles in academic databases, consulting textbooks, or conducting experiments. By combining the information from Gemini with other sources, students can ensure that their writing is based on the most accurate and up-to-date information available.
I think Gemini gives sound advice here. Use it.
As you said in your post, Bullshit research is certainly what many students fall back on. Whether it’s because of lack of motivation and care or they just can’t be asked to put in some sort of effort into an assignment, it becomes a habit that’s hard to break. Because it’s easy, and I myself have done this several times for various reasons. But secondary research is usually my go to, learning from the primary researches say is more beneficial to me for most scenarios. And I can appreciate you saying you’re not knocking social media or television opinions and gossip, however, far too much “fake news” and other inaccurate statements are out there today. It’s almost useless relying on them for research.
ReplyDeleteSecondary research is what most professors expect of their undergraduate students – in large part because they don't think undergrads know enough to do primary research. I think undergrads can do primary research, but it is more difficult and more time consuming than secondary research.
DeleteBernoff’s distinction between primary, secondary, and “bullshit” research serves as a great framework for understanding the depth expected in different types of research. For college students, the reminder to avoid “crappy” sources and instead seek reliable, unbiased information from libraries, academic databases, or even Google Scholar is needed. Bernoff’s point about Google’s algorithm reflecting the searcher’s biases shows an important issue: students often fall into the trap of confirmation bias without even realizing it, as Google tailors results based on users’ browsing history and interests. This serves as a warning that academic research should not be driven by ease or alignment with preconceived notions, but rather by credible, balanced sources. By mentioning the CRAP test (Currency, Reliability, Authority, Purpose), the author reinforces the need to evaluate all sources, including those generated by AI, and avoid over-reliance on these tools as a substitute for traditional research methods. Gemini’s ability to access current data makes it more suitable for recent information, but even this AI should be supplemented with scholarly sources to meet academic standards. Instructors view academic research as a process of learning how to gather, evaluate, and synthesize information, a much-needed skill in any professional context. This importance on understanding the credibility of sources, tied with the practical advice on navigating Google and AI resources, makes this a valuable guide for students striving to improve the quality of their research and achieve academic success.
ReplyDeleteI appreciate your confidence in sources such as academic libraries and databases; however, even those sources are biased, as numerous studies have shown – studies which may themselves be biased. Bias is extremely difficult to control. One of the best things you can do is show your sources, which is a bit like playing cards with your hand exposed. You should always be suspicious of those who don't reveal their sources. (I'll cite myself to support that claim).
DeleteAs I said previously I am very new to using AI. I knew what it was before taking this class, but had never used it. I did not realized it was "trained" and couldn't find the most up to date information. I do like that you encourage the use of AI but at the same time caution that it isn't always accurate. I think the CRAP test is a good method of checking your sources for quality. Like you I think Wikipedia passes the CRAP test but because so many other professors don't think it is a good source I have never used it in a paper. I do like using Wikipedia to occasionally find sources by reading about the subject I am writing on and as I am reading on Wikipedia I will follow their sources from the article and use the information from those sources.
ReplyDeleteMany Wikipedia articles are BS, and one of the first clues is that they include little or no sources for their information. Don't use Wiki articles that don't show their sources, as I mentioned in my comment above.
ReplyDeleteIt's essential for them to understand the difference between primary and secondary research. Bernoff's insights are particularly valuable because they highlight the importance of credibility and unbiased reporting, which are crucial for academic research.
ReplyDeleteAdding thoughts on using AI like ChatGPT and Gemini is a smart move. These tools can assist in gathering information and generating ideas, but it's important to remind students to critically evaluate the sources and ensure the data is reliable. AI can be a helpful starting point, but it shouldn't replace thorough, critical research practices.
I think students will benefit a lot from this discussion, especially as they prepare to enter the professional world where these skills are indispensable.
Accurate research is crucial because it forms the foundation of any credible and persuasive piece of writing. When you base your arguments on well-researched data, you enhance the reliability and validity of your work. This not only strengthens your arguments but also builds trust with your readers. Inaccurate or poorly researched information can undermine your credibility and lead to misinformation.
ReplyDeleteToo many students, and people around the world in general, center their beliefs in inaccurate "BS" research from sources that either don't exist or are made solely for biased reasons. Whether it's because they just don't know, don't care, or if they wholeheartedly believe in the misinformation they spread depends on the person, but all three are the same amount of dangerous in the long-term.
ReplyDeleteBullshit research is becoming more prevalent due to search engines using AI to filter what websites pop up first, leading to uninformative and disingenuous sources. It is possible to remedy this by using databases or libraries used by colleges and government agencies, but even these may sometimes not be enough when it comes to finding primary sources.
ReplyDeleteNot only is it refreshing to have the leader of the class to embrace technology, It is actually applicable to working life outside of academic world. What I have learned from this class has helped me in my everyday work. I appreciate the description of how to complete meaning full research and write something that is hopefully worth reading.
ReplyDeleteIts nice to have a teacher that truly understands technology and understand its here to help students not hurt them
ReplyDeleteAfter viewing this post, I admire how knowledgeable you are about technology and how most of the research that is performed today is becoming more AI generated. I feel like is pro's and con's to that. But you are teaching us the upside to it And yes BS research AND articles are def in its one site that we have been using for years and that is either wikipedia or any social media sights (sorry to cut this short but my laptop keyboard is messing up really bad)
ReplyDeleteI love how knowledgeable you are when it comes to using AI and how you encourage us to use it. Before this class I knew what AI was but never really used it because everyone always used to talk about them in a negative light. I'm still learning how to use AI and be successful for my own benefit, but this class is definitely helping with that.
ReplyDeleteThis is a very good post telling me about the different types of research. I agree that Gemini gives "Bullshit research" and I even have a hard time finding a good source after asking Gemini for 15 of them. I am still not sold on Ai so I just look up my topic and do the crap test to find a source and taking any .com with a grain of salt.
ReplyDeleteOften, bs sources are used without knowing that they are unreliable, biased sources, Wikipedia being the most common example many people trust in. As myself, I used to cite information from Wikipedia when I was younger, not knowing how inaccurate the information may be. Now that I have learned how to locate creditable sources, it made it easier for me to avoid those bs sources in my academic writings. It's very important to know the differences of using a creditable source and a bs source.
ReplyDelete