in real life, we wouldn't be using missiles anyway. we'd probably use lasers or other radiation
Missles would be extremely effective in space combat. Aside from causing the usual hull breeches, you can modify them to be seekers to eliminate the need of aiming and causing stray space bullets to go afloat. Shaped charges could be used to penetrate armour, create breeches, and minimize shrapnel: much like Anti-Tank HEAT shells. If you go the way of High Explosives, and lots of it, almost all of the energy of the shockwaves would be conducted through the materials of the victim vessel itself (as well as a tiny bit through the gas of the explosives). There isn't any air for the energy to be dissipated in, so it all gets thrown into the metal that it's hitting. This would cause massive amounts of spalling and damage to the internals of the vessel, as well as leaving a fairly big hole. There is a huge risk of "rouge shrapnel" in that case however, which could cause major problems down the road.
Lasers would mainly transfer heat to the medium it's pointed at, and when you're pointing at a big metal tub in space, the tub will be able to take in a lot of that heat. As you probably know, metal is super good at conducting heat, and the whole vessel will act like a giant heatsink. While having the advantage of being a lot more difficult to see and avoid (unless you're a big space cat), they probably wouldn't work all that well against a bigger vessel. They would be very helpful against active defense against missles however, as the target is a lot smaller, can't absorb as much heat energy, and also carries a payload sensitive to said heat. The laser pretty much cooks the missle, causing it to prematurely explode, which is fantastic; except until the missle turns into a very fast moving fragmentation grenade. While being hit by smaller things is usually a good thing, now there are tons of tiny little shards of metal and plastic with lots of kinetic energy heading towards their target, thanks to the fact that there is not air to absorb that energy, nor the energy from the explosion. If you eliminate the missle before it's actually heading towards you, like if it was course correcting, this isn't really a problem however.
The problem of shrapnel in space is huge though, as our own ISS has to deal with that problem. Space trash flying around at thousands of kilometres per hour is a pretty scary thing.
When something the size of a fleck of paint can do this to a shuttle window:
Imagine only what happens when that fleck is replaced by large bullets or fragments of metal. Bigger things will transfer more energy, only causing more problems.
Shaped charges could help alleviate this problem a lot, which is good: but so could large shells made of lead.
Take for example, the FV4005. Armed with a very large 183mm, or 7.2 inch cannon, this self propelled gun was designed to destroy cold-war soviet tanks in a single shot.
The thing pretty much (would've) destroyed the tanks by throwing a giant shell of energy at them, causing them to just bend and crumple into uselessness, by having the engine, transmission, suspension, or ammunition completely destroyed through deformation. also the crewmembers vital to having the tank function might get a little bit hurt.
Going off the weight and velocity of a 7.2 inch howitzer using a MK6. shell and
a predicted value of armour entry from this calculator, the shell could go through 173mm of armour.
Quite enough to punch through the IS-3's at the time, but also not enough to go completely through the vehicle. (Even on a non-penetrating hit, the amount of energy deforming the armour could cause huge amounts of cracking and structural failure as well).
This would probably be the most effective way to damage a space ship as well. Shoot a huge lead shell at it, letting it cause a massive hull breech, damaging and rendering internal modules (and people) deformed and useless, while also getting stuck somewhere inside the vessel.
Shooting a lead shells keeps the projectile soft, allowing it to transfer it's energy better, causing more damage, while also letting it deform hitting the last wall on the way out, preventing it from becoming dangerous space junk. Large shells also help prevent self-repair. Lots of plane have those "self repairing fuel tanks", which pretty much fill the bullet hole with some layer of jello or something. It works great with small bullets, but if there was a 7 inch hole in your wing, it might not work.
This big shell thing is also the most effective way to damage craft in SE as well. Those gravity guns that shoot rocks at super fast speeds can punch through almost any outer layers of armour while getting lodged inside the ship.
One helpful thing as well is if a large-calibre shell hits a spacecraft, most of the shrapnel will be propelled into the modules inside the ship, getting stuck in there instead of wandering about through space. If the big shell manages to go through the vessel, it will be very dangerous if it ever connects with something in the future for sure, but it also is a lot bigger than a bunch of paint flecks, allowing it to be tracked and avoided (assuming sensors or detection ever gets that good).
So really, the most practical space combat vessel is this:
Big guns, heavy armour, and a huge assortment of radar scanners and sensors to help avoid any incoming, rouge shells.
Being big also helps minimize overall damage to the vessel. Sure if something pokes a big hole in the ship, that's bad, but if your ship has thousands of rooms you can just close off, that breech becomes a lot less damaging. As well, it helps minimize the chance of shells going through the entire vessel.
It's a big trend as well: (click for big)
Going off the ships from Halo as well (at least the human ones), they seem to follow a fairly realistic progression in technology.
Vehicles still have wheels, spaceships are NUCLEAR POOOWEREEED, and the use traditional missles, guns, and magnetic rail guns.
All of these things would work very well against enemy ships, and (when the enemies dont have magic force fields), they do,
especially the rail guns.Essentially, big calibred, moderately penetrating guns, or shaped warheads will probably be the most effective at taking out space ships.
Although most of it is probably just general speculation because it hasn't happened all that much.
However, undeniably, the best way to kill of a space ship is just to make sure it runs out of fuel prematurely.
