Breaking the fourth wall is when the characters in a work of fiction make nods to the fact that they're in works of fiction. They do it all the time in Animaniacs, complaining about the episode's screenplay and animators, and making references to the audience.
It can also mean looking directly at the camera in a movie or audience in a play, which in theater is taboo, like saying the name of "That Scottish Play" in a theater.