Thesify.ai: Scholarly Writing in the Age of A.I.

[00:01] Announcer:

Welcome to Principal Center Radio, helping you build capacity for instructional leadership. Here's your host, Director of the Principal Center, Dr. Justin Bader. Welcome everyone to Principal Center Radio.

[00:13] SPEAKER_00:

I'm your host, Justin Bader, and I'm honored to welcome to the program, Dr. Mark Oliver-Gvaltig. Mark has a PhD in physics, but quickly switched to neuroscience to understand how our brains process information. Before co-founding his company, Thesify, Mark was senior group leader at the, I will say this incorrectly, but the Ecole Polytechnique Fédérale de Lausanne, and before that, a principal scientist at the Honda Research Institute Europe in Germany. And he's joining us from Switzerland today to talk about Thesify, an AI-powered app for scholarly writing.

[00:50] Announcer:

And now, our feature presentation.

[00:52] SPEAKER_00:

Mark, welcome to Principal Center Radio. Thanks for having me on the show. Well, I'm excited to talk about the work that you're doing and kind of how you got into that because it's focused specifically on academic writing, which is, I think, a big concern for educators worldwide. We are seeing, starting in the very early years of of schooling, perhaps students relying inappropriately on artificial intelligence. And you have built Thesify with those concerns in mind. And I wonder if you could take us into just a little bit of the story of how you came into this and what some of your goals were in building this app.

[01:34] SPEAKER_01:

Well, it took us actually quite a while to figure out what to do with AI in this space. We originally came from the area of scientific publishing, in fact. We were working with a different company before that with a scientific publisher, and we quickly realized that there is a big integrity concern across basically all strata of writing where not only fabricated content is being submitted, but also inadvertent plagiarism that goes with it. There's two types, of course. One is you're passing off something that is not your own as your own. But also, AI is likely to actually copy verbatim snippets of other people's work as well.

[02:24]

So that's the second source of plagiarism that might come in. And with that in mind, we figured out, OK, AI is such a great technology, it can certainly tell you how to do things very well. So if I think about my journey as a writer, it was always my professor who had to spend a lot of time revising my work and giving me feedback that actually formed me as a writer. And typically, you can only do that professionally. once or twice per manuscript. But with AI, you can do it as often as you like and as in deep as you want to have it.

[03:08]

So it's actually a great opportunity for very personalized writing instruction rather than text generation.

[03:19] SPEAKER_00:

Right. And I think with text generation, You know, there's the issue of plagiarism, as you mentioned. There's the issue of false citations that don't really exist. You know, papers that have maybe real authors, but the paper doesn't exist or, you know, there's some sort of information that's incorrect there. So rather than generating the writing, your goal with Thesify was to be kind of a writing coach or a writing tutor and help improve students' academic writing. Is that right?

[03:46] SPEAKER_01:

Exactly. So what we do is we ingest a manuscript that the user has written, and we basically review it like a reader, a reviewer, or a professor would. So we take instructions of what a student is supposed to write. So if as a student you have a writing prompt from your professor or your teacher, You give us that writing prompt along with what you have written. And then you get an in-depth analysis. And it's broken down into several rubrics.

[04:27]

One is purpose. Do you actually write about what you're supposed to write about? What are the topics that you write about? What are counter arguments that you could give? We analyze the thesis statement that you have. So is it actually understandable what you're claiming?

[04:44]

And then use of evidence. Do you actually ground your claims, your statements, either with an argument or with an external reference? And this is the type of feedback that you get. And then you get a summary with concrete advice on how to revise your manuscript.

[05:05] SPEAKER_00:

So very parallel to the type of feedback that you would get from a professor, but may not be able to get from a professor in advance of submitting the assignment.

[05:14] SPEAKER_01:

In fact, we are working with professors from University of Warwick in the UK to develop the rubrics.

[05:23] SPEAKER_00:

Very nice. I know university professors have had quite a bit of disruption from ChatGPT in particular, and there are a lot of concerns that perhaps students are no longer doing their own thinking, no longer doing their own research, no longer doing their own writing. I think a lot of professors wish that we could just go back in time to that point where these tools did not exist at all and everybody had to do their own work or, you know, plagiarize, pay someone else. You know, in some ways, these problems are not new though, right? We've always had problems of students not doing their own work.

[06:00] SPEAKER_01:

Yeah, indeed. I mean, the pocket calculator was probably one of the first disruptions of this type where people were not doing their own math anymore. I think there's two lessons that we as educators have to learn from that. One is, of And how are we assessing what we want to teach in fact? So if we go back, let's say a hundred years, there was a lot of memorization learning by heart and people thought that's the way to go to just learn facts, but you don't actually learn very much doing that. And then people realized, okay, you have to focus more on how you do things But if you teach how to do things, how do you assess it?

[06:49]

And we take the artifact that comes out of the how as a proxy for assessing the how, if you know what I mean. So we take the essay, for example, as the proxy for the act of being able to, or the act of writing, of the skill of writing. Whereas what we want to know is, is our student able to understand a particular topic formulate it in their own word, formulate a thesis around it and defend it appropriately. This is what we actually want to know. Taking just the essay as a measure of whether that was achieved or not I think is no longer possible because apparently technology can take a shortcut here. And in my view the only way out of this is that we completely acknowledge that this technology exists

[07:44]

And take the students actually on a little journey that has two parts. One is you give them a writing prompt and you say, okay, take that prompt and let ChatGPT or whatever tool is available, write an essay for you. Analyze that essay and then ask yourself, okay, does it actually answer or does it actually reflect your own thinking? as a student, your own opinion. Because one thing that CHET-GPT does, if 20 people are writing the same essay with the same tool, you'll get approximately 20 times the same essay. There is no diversity.

[08:23]

There is no reflection of individual thought. And here I think there's a big chance to actually teach that If you want to express your own ideas, you have to learn to express yourself. There's no way around it. And you cannot have AI actually talk for you. It's like having AI listen to music for you. It doesn't really work.

[08:51]

The more you think about it, you're not taking a forklift to the weight room because it's not helping you.

[08:59] SPEAKER_00:

That's a great metaphor. And I think very much what college professors are concerned about, what K-12 educators are concerned about, that students will not benefit from the work that they're doing because they're not doing it themselves. And when it comes to the writing process, you said something that I think is critical, but often overlooked. And that is that we need to actually teach what we're asking students to do, because students are not born knowing how to do you know, research papers, how to write academic papers. They have to be taught. And often no one along the way sees themselves as fully responsible for doing that teaching.

[09:38]

We just expect, you know, students are going to come in able to do the type of writing that I require in this course. So students may have gone all the way through their academic career, having never had very much instruction in how to do academic writing. So your platform is designed to help with that, correct?

[09:57] SPEAKER_01:

Exactly. I mean, my experience as a research scientist, I've supervised a lot of master and PhD students in neuroscience, computational neuroscience, and all the students without exception that I've had has never had a formal writing training. That's very different from humanities and social sciences where there is a rigorous writing training. But the academic disadvantage that you have, in particular as a non-native speaker, if you've never had writing training, is tremendous. You have to write applications. You have to write grant proposals.

[10:41]

You have to write research papers. You have to write abstracts for conferences. And writing an abstract, for example, which in some conferences is as short as 500 words, is extremely difficult. And if you've never learned that, you will not succeed. You will simply not get your work accepted to any of the major venues. And that will just...

[11:09]

Propagate, I mean, as a non-native speaker on that end, you have to work twice as hard as a native speaker to be competitive.

[11:19] SPEAKER_00:

So talk to us a little bit about how that teaching happens. So you said the student might start by inputting the prompt or the assignment that they are required to produce and then practice. The app does not write for them, right? How does it teach the writing process or at what points would it teach? Do I as a user need to write something and put it into the app and then get feedback on it? Is that the idea?

[11:43] SPEAKER_01:

So eventually we want to have a complete writing coach that will tell you from end to end, basically how you would have to approach writing. But we decided to start with something that gives maximum value to start with. I mean, we are a startup. At one point we have to earn money. So we cannot wait for a long time to see how people would react to a writing coach. So we start with the formative feedback side.

[12:13]

So you give us a draft and it can be a rough draft or it can be more complete draft. And our assistant, which we call Theo, will tell you how to improve that draft. It will not write for you. We'll just tell you, okay, these are topics you could write about or you're Input prompt said, you want to write an essay about a given topic, then it will remind you whether you have actually talked about that topic or not, or whether you're off topic. That's the purpose part. It will also remind you, okay, here you claim that, I don't know, Santa Claus was a saint.

[12:54]

Please give us some evidence that shows that this is true. So it will guide you through things that you can improve. It will point you to locations in your manuscript. And it's learning by doing, essentially. And then you can come back. You revise your manuscript.

[13:13]

You can upload a new version and continue this process. So that's what we're doing. We also... show you references that you could cite.

[13:23]

So for example, you can highlight a statement and then say, okay, find me references. But here we actually not relying on GPT to make up something that might not exist. We actually have a curated database of 200 million, I think it is, publications that do exist.

[13:44] SPEAKER_00:

The citations are real. And if a student is not sure what references to cite, this can actually help them find those citations and read the sources and understand the sources.

[13:55] SPEAKER_01:

Yeah. So we give you a small summary of what is said in the paper. Sometimes it would just, I mean, what we do is we look for semantic similarity. This is something you can evaluate nowadays with AI. It doesn't actually mean that this particular statement is actually given there in affirmative or negative. You will actually have to read what you cite, but we help you find the candidates that you should read.

[14:22] SPEAKER_00:

Given the huge range of styles and content addressed in different academic specialties, how do you in one app cover that full range and help students produce writing that will be appropriate for their discipline?

[14:36] SPEAKER_01:

So I would say so far we are best suited for documents from the humanities and social sciences where you do not have formulas, equations, long derivations there. that contribute to the content of the message. I mean, on the one extreme end, mathematical papers, which sometimes have so little text that there is hardly any prose where you could go wrong. I mean, you have a mathematical formalism that says it all, the expert know how to read it and you don't need any prose. On the other end, you have, of course, fictional work, which we are also, I mean, you will get feedback on fictional work, but when the AI tells you that you have to provide evidence that Gandalf was a wizard, it's not something that you might be able to produce.

[15:37] SPEAKER_00:

Right, right. So there's an orientation toward being able to cite real sources and make statements of fact and make logical arguments that lends itself to academic writing.

[15:45] SPEAKER_01:

Predominantly texts that rest within themselves. For example, what I mean with that is also if you have experimental work, like from chemistry, physics, and so on, it's also difficult because it's very difficult for AI nowadays to actually judge whether the methods... that were used are actually appropriate or adequate for that particular question and whether the analysis was done correctly. I mean, this is not something we can do right now, but if you talk about social sciences, philosophy, marketing, business, law, and so on, all these sorts of texts, anything you've write in school, basically as well, essays of any type, there are, very well suited to be analyzed with EC5.

[16:38] SPEAKER_00:

And certainly for anyone who is in a graduate program right now, I think one of their major concerns with any kind of writing is that they need to be able to talk about it. They need to be able to defend it. They need to be able to explain what they meant, because if they can't, then of course there's going to be suspicion. Oh, you use ChatGPT to write this. You didn't do this work yourself. So the unskippable step of knowing your sources, knowing your argument, having thought through all of that.

[17:05]

I love the way you've built that in. And I want to clarify, this is not a sponsored episode. No money has changed hands here. This is not an advertisement. I'm just very interested in how you're approaching this because I know many of our listeners are in a graduate program themselves. They're doing this type of writing and want to do a good job.

[17:22]

Don't want to cheat, want to have good sources, but also want to do writing that they're proud of and perhaps have not had a lot of training in that, a lot of feedback in that. So definitely would love to hear from people if you use this, if you do some writing in this way.

[17:40] SPEAKER_01:

I find question for me very interesting. How would learning evolve or teaching evolve with these new tools being around. And I remember when I was studying physics, computers were just entering universities. And I think we were one of the first to actually use the computer to analyze data that we had and hand it in a lab report that was written on a computer. Before that, they were written by hand. And I remember that our professors were getting upset at that time.

[18:12]

And nowadays, if you hand in something by hand that's written by hand at university level, you will not even get great. I mean, it's simply no fun. I think something that is very easy to do and that doesn't actually require using our tool is you can instruct GPT or Gemini or any of the others to do a review for you. And I think it would be a very good exercise actually with students to develop a prompt that would actually do that. It's not very difficult. You just have to know beforehand what type of analysis the AI should actually do.

[18:55]

And then you could look at the output of that and actually put in a text that you have. I mean, there are lots of example texts could be from the class, could be from the literature and just try to understand how well does the AI actually understand what's being written here and how well is it able to give you formative feedback? And what would you have to tell the AI to actually give you the feedback? One challenge that we discovered, for example, is the AI will avoid by almost all means to call something bad. You can put in a very bad article, a very bad essay. It will not say, well, this essay is really bad.

[19:45]

It always tries to be positive, even though sometimes It's important that you say, okay, nice try, but no. And I think you have to take the new generation along with these tools and make them aware of what they can do, what they cannot do, and how you can actually use it to your advantage in the sense that you actually learn to become a good writer. Yeah, so I think it's a relatively easy exercise to do, but you learn a great deal from that. So when we develop a new feature, we actually do it on ChatGPT first. We just try to do it with a simple prompt. before we go into a more evolved machinery so that you always get the same output.

[20:37]

We also validate. There are actually lots of data sets out there of graded essays where teachers and professors have graded essays. There are data sets out there and we use that to calibrate our tool. You don't need that if you want to do it yourself, but you learn a lot in the process about writing.

[20:58] SPEAKER_00:

Yeah. And I think for any university professor, anyone who is instructing graduate students, we want that to be the goal, right? We don't want the paper itself to be the goal. We want the learning and the human development process to be the primary outcome, that we're actually developing good writers and not just getting a stack of papers that were perhaps written by that forklift in the gym, as you said.

[21:20] SPEAKER_01:

I mean, writing is such a I think, underestimated skill because it makes you a better communicator in life and wherever you are able to express yourself better than somebody who has never had to think about a sentence and how it's made up and what it actually conveys to the listener.

[21:43] SPEAKER_00:

the reader yeah absolutely and i will certainly say writing is the skill that has served me best as an educator as an author even landing my first job as a principal depended heavily on being able to write a cover letter being able to you know write emails and make contacts and express interest talk to us about the range of types of writing we talked about the range of different subject areas but you know in the course of an academic career students may be asked to write short discussion prompt items. They may be asked to write, you know, short papers, long papers, all the way up to say a PhD dissertation. How does Thesify handle different lengths of writing tasks?

[22:26] SPEAKER_01:

So we started originally with relatively short form essays, a few hundred, maybe two or three thousand words. And The reason is, this is by frequency the most written format. I mean, people write a few theses or long documents in their life, but they very often write short pieces. What we're currently working on is really to go to book format, that you have individual chapters and so on. We are rolling that out just now, basically in the coming weeks. This is what we're working on.

[23:03]

As an academic, of course, and this is also something we are working on, you have... Formats that differ widely in the way they have to be written. So on the one end, I mentioned that already are abstracts, which I think are the most difficult format to write because they are so short. And you have to, in that few hundred words, you have to manage to talk about the problem, the state of the art of the problem.

[23:34]

Why is it interesting? What have you done? And what are the results of what you've done? And that is a lot for a few hundred words. On the other hand, if you have a PhD dissertation, it's also very difficult not to get lost in reviewing other people's work. and not talking about what is it that you have actually contributed that's in my experience the biggest mistake that students make is that they're actually not talking about what they did in the past three years they talk about a lot of other stuff and for each of these types of documents we will eventually have separate specific analysis workflows.

[24:23]

Right now it's relatively homogeneous around essentially an essay or a thesis.

[24:33] SPEAKER_00:

So yeah, let's talk a little bit more about the faculty perspective, because we can't prevent students from using any type of AI tool. We may be able to catch them if they're doing it in an obvious way or submitting false citations and things like that. But at the end of the day, we want our students to use the tools that are available and still accomplish the learning outcomes, still do the work, do the research, do the learning. So what would some of your advice be for higher education faculty who are teaching these courses, who are teaching and assigning writing of this type to their students?

[25:09] SPEAKER_01:

I think university faculty is in a very hard position right now because many universities and schools have not actually released coherent guidelines for how to use and not use AI. there are still many universities who have not issued any guidelines. So the faculty itself is actually in a very unsecure situation. There are also compliance issues, because if you do not have an enterprise version of any of these chatbots, then whatever you upload is being used for training, as opposed to if you have an enterprise version, or in our case, we also provide we have a specific agreement that the material is confidential. So that is an extra obstacle that university faculty has.

[26:04]

So some universities actually deployed their own version of ChatGPT. I know that ETH Zurich, for example, has one. Here the EPFL is rolling run out. There's a few other universities. So there, my advice to the professors would be, Be ahead of your students and try out everything that they could possibly try out and include it in your curriculum. Because if you do that, you're also signaling to the students, guys, I'm well aware that this exists.

[26:39]

And I also know what these things can do and cannot do. Many universities have access to a tool like Turnitin, which claims to actually detect AI-generated content, but it has a false positive rate of around 4%. So if you upload 100 essays that are flagged as AI-generated, four of them will actually be written by the student. and not be written by AI. So one has to be very careful with using these tools to draw any disciplinary consequences. So it's a difficult situation.

[27:16]

I think educators, they have to understand what can you do with these tools and what can you not do and try to figure out how do you achieve the same learning outcome in the new situation with these new tools.

[27:32] SPEAKER_00:

Mark, I think I'm going to try this. I'm going to take a topic that I am currently interested in. I've read one review paper that made some claims and made some citations about that topic. And I'm going to give this a try. I'm not actually writing a research paper on this, but I do want to understand the studies that have been done, understand the research, and be able to talk about it intelligently. So just for that purpose, I'm going to give this a try.

[27:56]

I'll share my results online if I can. Mark, if people want to sign up for your app? Where's the best place for them to go online?

[28:06] SPEAKER_01:

So if they want to sign up for our app, they just need to go to thesify.ai. And there is a free plan that you can use. It gives you 50 credits per month. And that's usually enough to have one or two short papers reviewed every month. Otherwise, you can sign up for a monthly or a yearly plan, which is very affordable, actually.

[28:30]

It's like $5 a month or $30 per year.

[28:35] SPEAKER_00:

very reasonable and yeah, not a sponsored episode at all, but I think this is very important for all of us to be aware of, to be continually learning what tools are out there, what they can do, what they can't do. And yeah, I share your concerns about, you know, inaccurate plagiarism detectors or the tools that are being sold to universities, not always getting it right. So I think there's no substitute for our own personal experience in using these tools and I think, you know, being thoughtful about how we want our students to use them. So again, the app is Thesify, T-H-E-S-I-F-Y dot A-I. And Dr. Kovaltek, thank you so much for joining me on Principal Center Radio.

[29:12]

It's been a pleasure.

[29:13] SPEAKER_01:

Thanks, Justin, for having me.

[29:15] Announcer:

Thanks for listening to Principal Center Radio. For more great episodes, subscribe on our website at principalcenter.com slash radio.