For an entire semester, Georgia Tech computer science professor Ashok Goel hid the fact that his online teaching assistant “Jill Watson” was a robot.
Goel did not disclose the robot until final grades were handed out. None of the students figured it out.
Each year, students have an option of selecting an assistant and next year Goel will do the same, but names will change. Students will have to guess if they are dealing with a robot or not.
Jill Watson
Please consider Professor reveals to students that his assistant was an AI all along.
To help with his class this year, a Georgia Tech professor hired Jill Watson, a teaching assistant unlike any other in the world. Throughout the semester, she answered questions online for students, relieving the professor’s overworked teaching staff.
Students were amazed. “I feel like I am part of history because of Jill and this class!” wrote one in the class’s online forum. “Just when I wanted to nominate Jill Watson as an outstanding TA in the CIOS survey!” said another.
Goel and his teaching assistants receive more than 10,000 questions a semester from students on the course’s online forum. Sometimes the same questions are asked again and again. Last year he began to wonder if he could automate the burden of answering so many repetitive questions.
As Goel looked for a technology that could help, he settled on IBM Watson, which he had used for several other projects. Watson, an artificial intelligence system, was designed to answer questions, so it seemed like a strong fit.
To train the system to answer questions correctly, Goel fed it forum posts from the class’s previous semesters. This gave Jill an extensive background in common questions and how they should be answered.
Goel tested the system privately for months, having his teaching assistants examine whether Jill’s answers were correct. Initially the system struggled with similar questions such as “Where can I find assignment two?” and “When is assignment two due?” Goel tweaked the software, adding more layers of decision-making to it. Eventually Jill reached the point where its answers were good enough.
“I cannot create chaos in my classroom. Jill had to be almost as perfect as a human TA or I am,” Goel said.
The system is only allowed to answer questions if it calculates that it is 97 percent or more confident in its answer. Goel found that was the threshold at which he could guarantee the system was accurate.
There are many questions Jill can’t handle. Those questions were reserved for human teaching assistants.
Goodbye TAs
Jill will not eliminate the need for all TA’s just 97% of them.
The real test comes next year. Students will ask all kinds of questions in attempts to figure out if they are dealing with a robot or not.
“Michelle Tutelage” better be prepared for ridiculous questions, and so must the real TAs. Precisely consistent answers and tones may help reveal the robot. I suspect human answers may vary a bit more than robots even for identical questions.
Perhaps Goel has already figured that out, or perhaps he will make a program change after reading this.
Secondly, if the same TAs help each year, then all of the TAs’ names must be disguised. So, why not just randomly assign the TAs?
“A really fun thing in this class has been once students knew about Jill they were so motivated, so engaged. I’ve never seen this kind of motivation and engagement,” Goel said. “What a beautiful way of teaching artificial intelligence.”
This kind of program will eliminate the need for TAs but it will also lower the cost of education making it more affordable.
Goel is forming a business to bring the chatbot to the wider world of education. “To me this is a grand challenge,” Goel said. “Education is such a huge priority for the entire human race.”
Mike “Mish” Shedlock
The old way of answering the question “When is assignment two due?” was to have the answer pinned on a notice-board. Sorry, pinned on a passive two-dimensional AI assistant.
Doesn’t the syllabus cover when assignments are due?
.
.
.” To train the system to answer questions correctly, Goel fed it forum posts from the class’s previous semesters. This gave Jill an extensive background in common questions and how they should be answered. ”
In the old days they called this a FAQ.
.
.
.” Goel tested the system privately for months, having his teaching assistants examine whether Jill’s answers were correct. ”
.
In the old days it took one person one day to post your FAQ on a website.
.
.
.
But Jill searches the FAQ for me, saving my lazy ass the trouble.
I’m shocked the college hasn’t replaced all the TAs already.
Just imagine how effective they’ll be once they start teaching political correctness. They’ll have students finding micro-aggression’s and cultural appropriations everywhere!
What used to be a no-brainer faq lookup now needs an “Artificial Intelligence” TA?!
No wonder the human race is headed into the Garbage Pit !!!
Soon the only intelligence will be in the AITAs 🙂
“This kind of program will eliminate the need for TAs but it will also lower the cost of education making it more affordable.”
Well it won’t make education more affordable, but it will certainly increase pay for top administrators.
+1
Robot TAs can also read/write English. Big leg up!
“This kind of program will eliminate the need for TAs but it will also lower the cost of education making it more affordable. ”
No, it won’t. If the AI misses just one trigger warning, another 6 administrators and 2 grief counselors will have to be hired.
“Students will ask all kings of questions in attempts to figure out if they are dealing with a robot or not”
“Hi, xxxxx … will you meet me at The Varsity to go over syllabus? My treat.”
A robot would not make that mistake
education is becoming more and more worthless everyday. why pay?
why pay for a robot?
aren’t you trying to get someone to teach you in real life?
critical thought and initiative are totally erased all you have is a robot following a list.
so essentially the robot is teaching you to be like itself… splendid
Mish, I think you need a new TA “The real test comes next year. Students will ask all kings of questions in attempts to figure out if they are dealing with a robot or not.”
Ok that explains it–Mish Watsonlock!
Technology companies should just eliminate hiring Ga tech grads and hire the AI teaching assistant instead. Should be cheaper.
I’ll bet Jill Watson doesn’t need any *cough* “safe spaces”, either.
First the assistant, then eventually the teachers.
The logical next step will be for students to employ their own AI “robots” to take the class for them, apply for student loans, and payoff the loan debt. Will those teaching online courses be able to distinguish the robotic students from the human students? Will anyone care, as long as the money flows?
This could really revolutionize telecommuting. Those with the best AI robots could hold multiple jobs. We could all stay home, and have our AI robot slaves handle the jobs. The perfect economy: Nobody “working” but everybody having multiple jobs
Nice one Joel. We are already nearly there with robotic economic modelling determining policy. All those new student loans to employ robots will be a big boost to GDP keeping the system really happy for another generation or two. Meanwhile we can all get on enjoying life as freed humans just like our paleo ancestors. Sounds great to me.
1) Nearly every comment so far is cynical! Why? The system seems pretty cool to me. And if that is the way people prefer to get their answers, rather than riffling through 100 FAQs, or having to go to the faculty office notice board for that subject, then why not? There is the potential trap that adjusting the computer’s knowledge base for the next year might be tedious, and could even tend towards creating a kind of inertia against changing the course it deals with. If the computer deals with simple facts like due dates of assignments, that is no problem. They could easily be changed year on year.
2) ‘Jill will not eliminate the need for all TA’s just 97% of them.’ No. The computer needed over 97% confidence to answer a question. It might have over 97% confidence for all or none of the questions. Also, from the article, it seems it only took on the common questions. We don’t know what percentage of the 10,000 questions are common questions. In all, from the article, we have no idea what percentage of questions the computer answered. No problem with that, but let’s be careful about how we assess the impact of this innovation.
Ha Ha. No. 2. 97%. Assumed he was being facetious.
so tell me how many jobs this AI robot created at GT, cause tech creates jobs right?
Hey all, don’t underestimate this, I am getting a master’s of IT in this GA Tech program (top school btw) and this class on Knowledge Based AI was incredible and highly rated by the students. Jill answered complex questions in a human manner without prompting (though she was monitored). None of us realized that she was an AI and we’re fairly smart folks – honest!
Jill was powered by IBM Watson, the same program that Mish recently posted about being used by a Law firm as “Ross”.
Last but not least, this is an online program – that you get a FULLY accredited masters degree (can even walk with the campus students on grad day) for about 7k on line. GO EDUCATION DISRUPTION!!! 🙂 I think Mish might have posted about this OMSCS program at one time.
In an online class like this with many students you want the TA’s focusing on real value add and great to have someone like “Jill” to field the basics…
Robots can’t teach leadership since leadership requires active learning using human-human interaction.
Is this a successful Turing Test? And why am I the first to ask?
today TA’s – tomorrow Professors
Let me reinforce the point by noting that deep mind it’s basically fed millions of scenes to seed typical responses. Teachers of a single course cover the same materials as associates, and do so repeatedly year to year. So there’s plenty of material. Bottom line – education should be free for online AI taught courses.
This is being a little hyped in your article and comments. It will NOT put 97% of assistants out of work, nor is it a successful Turing test. It is merely a good FAQ search.
There were many questions Jill could not answer. It did not answer 97% of them. That was the threshold of confidence above which it was ALLOWED to answer. Presumably below this, it kept silent or merely referred it to a human.
Eventually we will all have the same basic knowledge, as we will have augmented brains that are part wetware and part hardware. Information and software will be downloaded directly, with no learning or teaching required. Emphasis for human development will focus on healthy bodies, creativity, learning experiences, genetic tweaking, and interconnectivity in a borg-like existence. You will be assimilated. Resistance is futile.
Speaking as an ex-math TA, I think this is a great idea. As an adjunct who’s not given any TAs, I’d really like to have one of those to answer my emails.
Also, it may cut down on the grad school glut, as more and more are enticed into academic grad programs b/c of the program’s need for TAs. And then once they have their degrees, they find academic jobs are mainly low-paying adjunct positions.