The peak of their diversity was during the first half of the Paleozoic, with many different shapes of shells from coiled to straight, then they began to decline when their relatives the ammonites and coleoids appeared and began to compete for similar ecological niches. Although a few groups of nautiloids survived through the end-Permian mass extinction, most of them had disappeared by the end of the Triassic, leaving just one major remaining lineage known as the Nautilina (or Nautilaceae).
Cymatoceratids such as Cymatoceras sakalavum here had shells with a ribbed texture. Living during the Early Cretaceous, about 112-109 million years ago, this particular species is known from Japan and Madagascar and could reach a shell diameter of over 15cm (6″).
Hercoglossids, meanwhile, were much more smooth in appearance, but both groups also had more complex undulating sutures between their internal chambers than modern nautiluses do.
These nautiluses made it through the end-Cretaceous mass extinction and had a brief period of renewed success, filling the ecological roles left vacant by the extinct ammonites. But by the end of the Oligocene (~23 mya) both the cymatoceratids and hercoglossids vanished, possibly unable to deal with cooling oceans and the evolution of new predators.
Some of the hercoglossids’ Cenozoic descendants, the aturiids, managed to last a little longer into the Early Pliocene (~5 mya) before another period of cooling seems to have finished them off. Past that point, all that was left of the once-massive nautiloid lineage were their cousins the nautilids, who gave rise to today’s few living representatives.
(It’s also worth noting that the classification of the cymatoceratids seems to be in flux right now. Some paleontologists currently don’t consider Cymatoceras itself to actually be part of the group, instead being a nautilid much closer related to modern nautiluses. If this is the case then the cymatoceratids may not have actually survived past the Late Cretaceous – but the Cymatoceras genus alone still counts as an “almost-living” fossil since its various species ranged from the Late Jurassic to the Late Oligocene.)