It is reported that ‘Umar b. Al-Khattāb – Allāh be pleased with him – said:

I have considered sending out men to the different regions to check on all those who are well off enough but have not made Hajj, and impose the jizyah (tax imposed on Ahl Al-Kitāb  in the Muslim state) on them; they are not Muslims, they are not Muslims.

It is also reported that he said:

If the people abandoned Hajj, we would fight them for it as we fight them for the prayer and zakāh.

It is also reported that he said:

Whoever dies, being well off enough but having never done Hajj, let him die as a Jew if he wishes or as a Christian.

It is reported that Al-Aswad b. Hilāl – Allāh have mercy on him – said to a freedman of his called Miqlāṣ:

If you died and had never made Hajj, I would not pray over you.

It is reported that Sa’īd b. Jubayr – Allāh have mercy on him – said:

If a neighbor of mine died without ever making Hajj, while being well off enough to do so, I would not pray over him.

It is reported that Mujāhid b. Rūmī said:

I asked Sa’īd b. Jubayr, ‘Abd Al-Raḥmān b. Abī Laylā and Ibn Ma’qal (‘Abdullah Al-Muzanī) about a man who died, being well off enough, but never having made Hajj. Ibn Abī Laylā said, “I hope that if his next of kin does Hajj on his behalf…” Sa’īd b. Jubayr said, “The Fire, the Fire.” ‘Abdullah b. Ma’qal said, “He died in a state of disobedience to Allāh.”

Al-Khallāl, Al-Sunnah 5:43-47, articles 1572-1576.