The original metaphor that got us the words "shell" and "kernel" in the first place, for components of an operating system, is too often forgotten. The metaphor is that an operating system is a seed or a nut. The "kernel" of the seed is the core of the operating system, providing operating system services to applications programs, which is surrounded by the "shell" of the seed that is what users see from the outside.
Some people want to tie "kernel" and "shell" down to be more specific than that.
But in truth there's a lot of variation across operating systems.
Not the least these variations is what constitutes a "shell", which can range from Solaris' sh through Netware's Console Command Interpreter to OS/2's Workplace Shell and Windows NT's Explorer; but there's also a lot of variance from one operating system to another in what is, and isn't, a part of a "kernel" too (which may or may not include disk I/O, for example).
It's best to remember that these terms are metaphors.
Forgetting the origins of these metaphors results in one mistaking the part for the whole and mistaking the kernel for the whole operating system. Being a "userland" program does not make the shell any less the outer part of the operating system, nor make it any less that which encloses the kernel. Linux is not an operating system, merely the kernel of one.
There are other terminologies.
One place where the nut/seed metaphor was fairly foreign was MS/PC/DR-DOS in its early years, which favoured terminology from CP/M.
Like CP/M before it, MS-DOS was divided into the basic disk operating system (BDOS), the basic input-output system (BIOS — not the system firmware, note), the command processor, and the housekeeping utilities.
The code was actually parcelled up into files this way (ibmdos.com, ibmbio.com. command.com et al.).
This is another example, in fact, of how the kernel/shell metaphor doesn't really cover or fully match up with the actual divisions in many operating systems.
In IBM mainframe terminology, the "kernel" in the seed/nut metaphor is called the control program. Other names include the supervisor, the monitor, the core, and the executive. "core" is another fairly obvious biological metaphor.
It is interesting that so too is another name. Harvey M. Deitel, in his book Operating Systems, calls the control program the nucleus, and says
The nucleus normally represents a small portion of the code of what is commonly thought to be the entire operating system [...]
The word "nucleus" comes from the Latin nucleus, and means the kernel of a nut (c.f. the Latin nux, which means "nut"). So even calling the control program "the nucleus" is in fact making this same operating-system-as-a-nut analogy. It's an analogy that is used for more than just operating systems. If you are familiar with chemistry, for example, you'll know that an atomic nucleus is surrounded by electrons in shells (until an atomic physicist comes along and explains that it's as poor a description of reality for atoms as it is for operating systems).
Deitel isn't alone, either. There are reams of Unix books explaining the Unix-as-a-nut metaphor, from the 1980s onwards. I recommend Urban's and Tiemann's Teach Yourself FreeBSD in 24 Hours (SAMS, ISBN 9780672324246), for the simple reason that on page 54 it has a diagram illustrating the operating-system-as-a-seed metaphor, and — surprisingly — is one of few books to do so.
The name "nucleus" is probably the older name; certainly predating Unix, which favours the kernel/shell terminology that today — probably in no small part thanks to Unix — is the more prevalent. Here's what's in Alan C. Shaw's 1974 book The logical design of operating systems (Prentice-Hall, ISBN 9780135401125) for example:
That part of the system which permanently resides in main storage has historically been called the nucleus. The nucleus will usually consist of a minimal set of primitives and processes for the management of processes, resources, and input-output.
Shaw has a diagram, too. ☺